An Affective User Interface Based on Facial Expression Recognition and Eye-Gaze Tracking

2005 ◽  
pp. 907-914 ◽  
Author(s):  
Soo-Mi Choi ◽  
Yong-Guk Kim
Keyword(s):  
User Interface ◽  
Facial Expression ◽  
Facial Expression Recognition ◽  
Eye Gaze ◽  
Expression Recognition ◽  
Gaze Tracking

scholarly journals Toward Continuous Social Phenotyping: Analyzing Gaze Patterns in an Emotion Recognition Task for Children With Autism Through Wearable Smart Glasses

10.2196/13810 ◽  
2020 ◽  
Vol 22 (4) ◽  
pp. e13810 ◽  
Author(s):  
Anish Nag ◽  
Nick Haber ◽  
Catalin Voss ◽  
Serena Tamura ◽  
Jena Daniels ◽  
...  
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expression Recognition ◽  
Eye Gaze ◽  
Recognition Task ◽  
Children With Autism ◽  
Expression Recognition ◽  
Eye Tracker ◽  
Gaze Tracking ◽  
Smart Glasses

Background Several studies have shown that facial attention differs in children with autism. Measuring eye gaze and emotion recognition in children with autism is challenging, as standard clinical assessments must be delivered in clinical settings by a trained clinician. Wearable technologies may be able to bring eye gaze and emotion recognition into natural social interactions and settings. Objective This study aimed to test: (1) the feasibility of tracking gaze using wearable smart glasses during a facial expression recognition task and (2) the ability of these gaze-tracking data, together with facial expression recognition responses, to distinguish children with autism from neurotypical controls (NCs). Methods We compared the eye gaze and emotion recognition patterns of 16 children with autism spectrum disorder (ASD) and 17 children without ASD via wearable smart glasses fitted with a custom eye tracker. Children identified static facial expressions of images presented on a computer screen along with nonsocial distractors while wearing Google Glass and the eye tracker. Faces were presented in three trials, during one of which children received feedback in the form of the correct classification. We employed hybrid human-labeling and computer vision–enabled methods for pupil tracking and world–gaze translation calibration. We analyzed the impact of gaze and emotion recognition features in a prediction task aiming to distinguish children with ASD from NC participants. Results Gaze and emotion recognition patterns enabled the training of a classifier that distinguished ASD and NC groups. However, it was unable to significantly outperform other classifiers that used only age and gender features, suggesting that further work is necessary to disentangle these effects. Conclusions Although wearable smart glasses show promise in identifying subtle differences in gaze tracking and emotion recognition patterns in children with and without ASD, the present form factor and data do not allow for these differences to be reliably exploited by machine learning systems. Resolving these challenges will be an important step toward continuous tracking of the ASD phenotype.

scholarly journals Toward Continuous Social Phenotyping: Analyzing Gaze Patterns in an Emotion Recognition Task for Children With Autism Through Wearable Smart Glasses (Preprint)

2019 ◽  
Author(s):  
Anish Nag ◽  
Nick Haber ◽  
Catalin Voss ◽  
Serena Tamura ◽  
Jena Daniels ◽  
...  
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expression Recognition ◽  
Eye Gaze ◽  
Recognition Task ◽  
Children With Autism ◽  
Expression Recognition ◽  
Eye Tracker ◽  
Gaze Tracking ◽  
Smart Glasses

BACKGROUND Several studies have shown that facial attention differs in children with autism. Measuring eye gaze and emotion recognition in children with autism is challenging, as standard clinical assessments must be delivered in clinical settings by a trained clinician. Wearable technologies may be able to bring eye gaze and emotion recognition into natural social interactions and settings. OBJECTIVE This study aimed to test: (1) the feasibility of tracking gaze using wearable smart glasses during a facial expression recognition task and (2) the ability of these gaze-tracking data, together with facial expression recognition responses, to distinguish children with autism from neurotypical controls (NCs). METHODS We compared the eye gaze and emotion recognition patterns of 16 children with autism spectrum disorder (ASD) and 17 children without ASD via wearable smart glasses fitted with a custom eye tracker. Children identified static facial expressions of images presented on a computer screen along with nonsocial distractors while wearing Google Glass and the eye tracker. Faces were presented in three trials, during one of which children received feedback in the form of the correct classification. We employed hybrid human-labeling and computer vision–enabled methods for pupil tracking and world–gaze translation calibration. We analyzed the impact of gaze and emotion recognition features in a prediction task aiming to distinguish children with ASD from NC participants. RESULTS Gaze and emotion recognition patterns enabled the training of a classifier that distinguished ASD and NC groups. However, it was unable to significantly outperform other classifiers that used only age and gender features, suggesting that further work is necessary to disentangle these effects. CONCLUSIONS Although wearable smart glasses show promise in identifying subtle differences in gaze tracking and emotion recognition patterns in children with and without ASD, the present form factor and data do not allow for these differences to be reliably exploited by machine learning systems. Resolving these challenges will be an important step toward continuous tracking of the ASD phenotype.

Facial Expression Analysis, Modeling and Synthesis

2009 ◽  
pp. 34-55 ◽  
Author(s):  
Christoph Bartneck ◽  
Michael J. Lyons
Keyword(s):  
Facial Expression ◽  
Eye Gaze ◽  
Human Robot Interaction ◽  
Information Modeling ◽  
Human Interaction ◽  
Expression Recognition ◽  
Emotional States ◽  
Human Face ◽  
Gaze Tracking ◽  
Facial Expression Analysis

The human face plays a central role in most forms of natural human interaction so we may expect that computational methods for analysis of facial information, modeling of internal emotional states, and methods for graphical synthesis of faces and facial expressions will play a growing role in human-computer and human-robot interaction. However, certain areas of face-based HCI, such as facial expression recognition and robotic facial display have lagged others, such as eye-gaze tracking, facial recognition, and conversational characters. Our goal in this paper is to review the situation in HCI with regards to the human face, and to discuss strategies, which could bring more slowly developing areas up to speed. In particular, we are proposing the “The Art of the Soluble” as a strategy forward and provide examples that successfully applied this strategy.

scholarly journals Real-time Facial Expression Recognition to Track Non-verbal Behaviors as Lie Indicators During Interview

2019 ◽  
Vol 1 (1) ◽  
pp. 25-31
Author(s):  
Arif Budi Setiawan ◽  
Kaspul Anwar ◽  
Laelatul Azizah ◽  
Adhi Prahara
Keyword(s):  
Facial Expression ◽  
Real Time ◽  
Facial Expression Recognition ◽  
Eye Gaze ◽  
Processing Method ◽  
Facial Features ◽  
Expression Recognition ◽  
Nonverbal Behaviors ◽  
Facial Landmark ◽  
Short Time

During interview, a psychologist should pay attention to every gesture and response, both verbal and nonverbal language/behaviors, made by the client. Psychologist certainly has limitation in recognizing every gesture and response that indicates a lie, especially in interpreting nonverbal behaviors that usually occurs in a short time. In this research, a real time facial expression recognition is proposed to track nonverbal behaviors to help psychologist keep informed about the change of facial expression that indicate a lie. The method tracks eye gaze, wrinkles on the forehead, and false smile using combination of face detection and facial landmark recognition to find the facial features and image processing method to track the nonverbal behaviors in facial features. Every nonverbal behavior is recorded and logged according to the video timeline to assist the psychologist analyze the behavior of the client. The result of tracking nonverbal behaviors of face is accurate and expected to be useful assistant for the psychologists.

scholarly journals Facial Expression Recognition of Nonlinear Facial Variations Using Deep Locality De-Expression Residue Learning in the Wild

Electronics ◽  
2019 ◽  
Vol 8 (12) ◽  
pp. 1487 ◽  
Author(s):  
Asad Ullah ◽  
Jing Wang ◽  
M. Shahid Anwar ◽  
Usman Ahmad ◽  
Uzair Saeed ◽  
...  
Keyword(s):  
Facial Expression ◽  
Real World ◽  
Facial Expression Recognition ◽  
Eye Gaze ◽  
Input Image ◽  
Training Data ◽  
Head Pose Estimation ◽  
Expression Recognition ◽  
Action Unit ◽  
In The Wild

Automatic facial expression recognition is an emerging field. Moreover, the interest has been increased with the transition from laboratory-controlled conditions to in the wild scenarios. Most of the research has been done over nonoccluded faces under the constrained environment, while automatic facial expression is less understood/implemented for partial occlusion in the real world conditions. Apart from that, our research aims to tackle the issues of overfitting (caused by the shortage of adequate training data) and to alleviate the expression-unrelated/intraclass/nonlinear facial variations, such as head pose estimation, eye gaze estimation, intensity and microexpressions. In our research, we control the magnitude of each Action Unit (AU) and combine several of the Action Unit combinations to leverage learning from the generative and discriminative representations for automatic FER. We have also addressed the problem of diversification of expressions from lab controlled to real-world scenarios from our cross-database study and proposed a model for enhancement of the discriminative power of deep features while increasing the interclass scatters, by preserving the locality closeness. Furthermore, facial expression consists of an expressive component as well as neutral component, so we proposed a generative model which is capable of generating neutral expression from an input image using cGAN. The expressive component is filtered and passed to the intermediate layers and the process is called De-expression Residue Learning. The residue in the intermediate/middle layers is very important for learning through expressive components. Finally, we validate the effectiveness of our method (DLP-DeRL) through qualitative and quantitative experimental results using four databases. Our method is more accurate and robust, and outperforms all the existing methods (hand crafted features and deep learning) while dealing the images in the wild.

Sign in / Sign up

Export Citation Format

Share Document