Multimodal emotion recognition by combining physiological signals and facial expressions: A preliminary study

2012 ◽  
Author(s):  
J. Kortelainen ◽  
S. Tiinanen ◽  
Xiaohua Huang ◽  
Xiaobai Li ◽  
S. Laukka ◽  
...  
Keyword(s):  
Emotion Recognition ◽  
Facial Expressions ◽  
Physiological Signals ◽  
Preliminary Study ◽  
Multimodal Emotion Recognition

A multimodal emotion recognition method based on facial expressions and electroencephalography

2021 ◽  
Vol 70 ◽  
pp. 103029
Author(s):  
Ying Tan ◽  
Zhe Sun ◽  
Feng Duan ◽  
Jordi Solé-Casals ◽  
Cesar F. Caiafa
Keyword(s):  
Emotion Recognition ◽  
Facial Expressions ◽  
Recognition Method ◽  
Multimodal Emotion Recognition

scholarly journals A Review of Emotion Recognition Based on EEG using DEAP Dataset

2021 ◽  
pp. 298-303
Author(s):  
Rama Chaudhary ◽  
Ram Avtar Jaswal
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expressions ◽  
Physiological Signals ◽  
Emotional States ◽  
Modern Time ◽  
Human Machine Interaction ◽  
Electrocardiogram Ecg ◽  
Electroencephalogram Eeg ◽  
Machine Interaction

In modern time, the human-machine interaction technology has been developed so much for recognizing human emotional states depending on physiological signals. The emotional states of human can be recognized by using facial expressions, but sometimes it doesn’t give accurate results. For example, if we detect the accuracy of facial expression of sad person, then it will not give fully satisfied result because sad expression also include frustration, irritation, anger, etc. therefore, it will not be possible to determine the particular expression. Therefore, emotion recognition using Electroencephalogram (EEG), Electrocardiogram (ECG) has gained so much attraction because these are based on brain and heart signals respectively. So, after analyzing all the factors, it is decided to recognize emotional states based on EEG using DEAP Dataset. So that, the better accuracy can be achieved.

scholarly journals Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition

2017 ◽  
Vol 2017 ◽  
pp. 1-8 ◽  
Author(s):  
Yongrui Huang ◽  
Jianhao Yang ◽  
Pengkai Liao ◽  
Jiahui Pan
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expressions ◽  
Multimodal Fusion ◽  
Support Vector ◽  
Basic Emotion ◽  
Input Signals ◽  
Vector Machines ◽  
Fusion Methods ◽  
Multimodal Emotion Recognition

This paper proposes two multimodal fusion methods between brain and peripheral signals for emotion recognition. The input signals are electroencephalogram and facial expression. The stimuli are based on a subset of movie clips that correspond to four specific areas of valance-arousal emotional space (happiness, neutral, sadness, and fear). For facial expression detection, four basic emotion states (happiness, neutral, sadness, and fear) are detected by a neural network classifier. For EEG detection, four basic emotion states and three emotion intensity levels (strong, ordinary, and weak) are detected by two support vector machines (SVM) classifiers, respectively. Emotion recognition is based on two decision-level fusion methods of both EEG and facial expression detections by using a sum rule or a production rule. Twenty healthy subjects attended two experiments. The results show that the accuracies of two multimodal fusion detections are 81.25% and 82.75%, respectively, which are both higher than that of facial expression (74.38%) or EEG detection (66.88%). The combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources.

Fine-grained emotion recognition: fusion of physiological signals and facial expressions on spontaneous emotion corpus

2020 ◽  
Vol 35 (3) ◽  
pp. 162
Author(s):  
Feri Setiawan ◽  
Aria Ghora Prabono ◽  
Sunder Ali Khowaja ◽  
Wangsoo Kim ◽  
Kyoungsoo Park ◽  
...  
Keyword(s):  
Emotion Recognition ◽  
Facial Expressions ◽  
Physiological Signals ◽  
Fine Grained

Fine-grained emotion recognition: fusion of physiological signals and facial expressions on spontaneous emotion corpus

2020 ◽  
Vol 35 (3) ◽  
pp. 162
Author(s):  
Seok Lyong Lee ◽  
Jin Pyo Hong ◽  
Wangsoo Kim ◽  
Kyoungsoo Park ◽  
Bernardo Nugroho Yahya ◽  
...  
Keyword(s):  
Emotion Recognition ◽  
Facial Expressions ◽  
Physiological Signals ◽  
Fine Grained

scholarly journals FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network

Sensors ◽  
2020 ◽  
Vol 20 (18) ◽  
pp. 5328
Author(s):  
Clarence Tan ◽  
Gerardo Ceballos ◽  
Nikola Kasabov ◽  
Narayan Puthanmadam Subramaniyam
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Emotion Recognition ◽  
Skin Temperature ◽  
Facial Expressions ◽  
Affective Computing ◽  
Physiological Signals ◽  
Recognition Problem ◽  
Learning Methods ◽  
Multimodal Data

Using multimodal signals to solve the problem of emotion recognition is one of the emerging trends in affective computing. Several studies have utilized state of the art deep learning methods and combined physiological signals, such as the electrocardiogram (EEG), electroencephalogram (ECG), skin temperature, along with facial expressions, voice, posture to name a few, in order to classify emotions. Spiking neural networks (SNNs) represent the third generation of neural networks and employ biologically plausible models of neurons. SNNs have been shown to handle Spatio-temporal data, which is essentially the nature of the data encountered in emotion recognition problem, in an efficient manner. In this work, for the first time, we propose the application of SNNs in order to solve the emotion recognition problem with the multimodal dataset. Specifically, we use the NeuCube framework, which employs an evolving SNN architecture to classify emotional valence and evaluate the performance of our approach on the MAHNOB-HCI dataset. The multimodal data used in our work consists of facial expressions along with physiological signals such as ECG, skin temperature, skin conductance, respiration signal, mouth length, and pupil size. We perform classification under the Leave-One-Subject-Out (LOSO) cross-validation mode. Our results show that the proposed approach achieves an accuracy of 73.15% for classifying binary valence when applying feature-level fusion, which is comparable to other deep learning methods. We achieve this accuracy even without using EEG, which other deep learning methods have relied on to achieve this level of accuracy. In conclusion, we have demonstrated that the SNN can be successfully used for solving the emotion recognition problem with multimodal data and also provide directions for future research utilizing SNN for Affective computing. In addition to the good accuracy, the SNN recognition system is requires incrementally trainable on new data in an adaptive way. It only one pass training, which makes it suitable for practical and on-line applications. These features are not manifested in other methods for this problem.

Sign in / Sign up

Export Citation Format

Share Document