Bimodal System for Emotion Recognition from Facial Expressions and Physiological Signals Using Feature-Level Fusion

2011 ◽  
Author(s):  
F. Abdat ◽  
C. Maaoui ◽  
A. Pruski
Keyword(s):  
Emotion Recognition ◽  
Facial Expressions ◽  
Physiological Signals ◽  
Feature Level Fusion ◽  
Level Fusion

scholarly journals Audiovisual Emotion Recognition Using Entropy-estimation-based Multimodal Information Fusion

2021 ◽  
Author(s):  
Zhibing Xie
Keyword(s):  
Human Computer Interaction ◽  
Emotion Recognition ◽  
Information Fusion ◽  
Score Level Fusion ◽  
Information Theoretic ◽  
Multimodal Information Fusion ◽  
Multimodal Information ◽  
Feature Level Fusion ◽  
Level Fusion ◽  
Computer Interaction

Understanding human emotional states is indispensable for our daily interaction, and we can enjoy more natural and friendly human computer interaction (HCI) experience by fully utilizing human’s affective states. In the application of emotion recognition, multimodal information fusion is widely used to discover the relationships of multiple information sources and make joint use of a number of channels, such as speech, facial expression, gesture and physiological processes. This thesis proposes a new framework of emotion recognition using information fusion based on the estimation of information entropy. The novel techniques of information theoretic learning are applied to feature level fusion and score level fusion. The most critical issues for feature level fusion are feature transformation and dimensionality reduction. The existing methods depend on the second order statistics, which is only optimal for Gaussian-like distributions. By incorporating information theoretic tools, a new feature level fusion method based on kernel entropy component analysis is proposed. For score level fusion, most previous methods focus on predefined rule based approaches, which are usually heuristic. In this thesis, a connection between information fusion and maximum correntropy criterion is established for effective score level fusion. Feature level fusion and score level fusion methods are then combined to introduce a two-stage fusion platform. The proposed methods are applied to audiovisual emotion recognition, and their effectiveness is evaluated by experiments on two publicly available audiovisual emotion databases. The experimental results demonstrate that the proposed algorithms achieve improved performance in comparison with the existing methods. The work of this thesis offers a promising direction to design more advanced emotion recognition systems based on multimodal information fusion and has great significance to the development of intelligent human computer interaction systems.

scholarly journals Intelligent Music Player Based on Emotions

2020 ◽  
Vol 9 (3) ◽  
pp. 525-528
Keyword(s):  
Facial Expressions ◽  
Feature Vector ◽  
Emotional Experience ◽  
Head Movements ◽  
Cascade Classifier ◽  
Feature Level Fusion ◽  
Web Camera ◽  
Music Player ◽  
Combined Feature ◽  
Level Fusion

This project presents a system to automatically detect emotional dichotomy and mixed emotional experience using a Linux based system. Facial expressions, head movements and facial gestures were captured from pictorial input in order to create attributes such as distance, coordinates and movement of tracked points. Web camera is used to extract spectral attributes. Features are calculated using Fisherface algorithm. Emotion detected by cascade classifier and feature level fusion was used to create a combined feature vector. Live actions of user are to be used for recording emotions. As per calculated result system will play songs and display books list.

scholarly journals EMOTION ANALYSIS USING SIGNAL AND IMAGE PROCESSING APPROACH BY IMPLEMENTING DEEP NEURAL NETWORK

2021 ◽  
Vol 57 (2) ◽  
pp. 313-321
Author(s):  
S Shuma ◽  
◽  
T. Christy Bobby ◽  
S. Malathi ◽  
◽  
...  
Keyword(s):  
Emotion Recognition ◽  
Recognition Rate ◽  
Principal Component ◽  
Well Being ◽  
Recognition System ◽  
Autism Spectrum ◽  
Learning Networks ◽  
Human Communication ◽  
Feature Level Fusion ◽  
Level Fusion

Emotion recognition is important in human communication and to achieve a complete interaction between humans and machines. In medical applications, emotion recognition is used to assist the children with Autism Spectrum Disorder (ASD to improve their socio-emotional communication, helps doctors with diagnosis of diseases such as depression and dementia and also helps the caretakers of older patients to monitor their well-being. This paper discusses the application of feature level fusion of speech and facial expressions of different emotions such as neutral, happy, sad, angry, surprise, fearful and disgust. Also, to explore how best to build the deep learning networks to classify the emotions independently and jointly from these two modalities. VGG-model is utilized to extract features from facial images, and spectral features are extracted from speech signals. Further, feature level fusion technique is adopted to fuse the features extracted from the two modalities. Principal Component Analysis (PCA is implemented to choose the significant features. The proposed method achieved a maximum score of 90% on training set and 82% on validation set. The recognition rate in case of multimodal data improved greatly when compared to unimodal system. The multimodal system gave an improvement of 9% compared to the performance of the system based on speech. Thus, result shows that the proposed Multimodal Emotion Recognition (MER outperform the unimodal emotion recognition system.

scholarly journals A Review of Emotion Recognition Based on EEG using DEAP Dataset

2021 ◽  
pp. 298-303
Author(s):  
Rama Chaudhary ◽  
Ram Avtar Jaswal
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expressions ◽  
Physiological Signals ◽  
Emotional States ◽  
Modern Time ◽  
Human Machine Interaction ◽  
Electrocardiogram Ecg ◽  
Electroencephalogram Eeg ◽  
Machine Interaction

In modern time, the human-machine interaction technology has been developed so much for recognizing human emotional states depending on physiological signals. The emotional states of human can be recognized by using facial expressions, but sometimes it doesn’t give accurate results. For example, if we detect the accuracy of facial expression of sad person, then it will not give fully satisfied result because sad expression also include frustration, irritation, anger, etc. therefore, it will not be possible to determine the particular expression. Therefore, emotion recognition using Electroencephalogram (EEG), Electrocardiogram (ECG) has gained so much attraction because these are based on brain and heart signals respectively. So, after analyzing all the factors, it is decided to recognize emotional states based on EEG using DEAP Dataset. So that, the better accuracy can be achieved.

scholarly journals Audiovisual Emotion Recognition Using Entropy-estimation-based Multimodal Information Fusion

2021 ◽  
Author(s):  
Zhibing Xie
Keyword(s):  
Human Computer Interaction ◽  
Emotion Recognition ◽  
Information Fusion ◽  
Score Level Fusion ◽  
Information Theoretic ◽  
Multimodal Information Fusion ◽  
Multimodal Information ◽  
Feature Level Fusion ◽  
Level Fusion ◽  
Computer Interaction

Understanding human emotional states is indispensable for our daily interaction, and we can enjoy more natural and friendly human computer interaction (HCI) experience by fully utilizing human’s affective states. In the application of emotion recognition, multimodal information fusion is widely used to discover the relationships of multiple information sources and make joint use of a number of channels, such as speech, facial expression, gesture and physiological processes. This thesis proposes a new framework of emotion recognition using information fusion based on the estimation of information entropy. The novel techniques of information theoretic learning are applied to feature level fusion and score level fusion. The most critical issues for feature level fusion are feature transformation and dimensionality reduction. The existing methods depend on the second order statistics, which is only optimal for Gaussian-like distributions. By incorporating information theoretic tools, a new feature level fusion method based on kernel entropy component analysis is proposed. For score level fusion, most previous methods focus on predefined rule based approaches, which are usually heuristic. In this thesis, a connection between information fusion and maximum correntropy criterion is established for effective score level fusion. Feature level fusion and score level fusion methods are then combined to introduce a two-stage fusion platform. The proposed methods are applied to audiovisual emotion recognition, and their effectiveness is evaluated by experiments on two publicly available audiovisual emotion databases. The experimental results demonstrate that the proposed algorithms achieve improved performance in comparison with the existing methods. The work of this thesis offers a promising direction to design more advanced emotion recognition systems based on multimodal information fusion and has great significance to the development of intelligent human computer interaction systems.

Sign in / Sign up

Export Citation Format

Share Document