scholarly journals Using Brain Activity Patterns to Differentiate Real and Virtual Attended Targets during Augmented Reality Scenarios

Information ◽  
2021 ◽  
Vol 12 (6) ◽  
pp. 226
Author(s):  
Lisa-Marie Vortmann ◽  
Leonid Schwenke ◽  
Felix Putze
Keyword(s):  
Augmented Reality ◽  
Eye Tracking ◽  
Brain Activity ◽  
Activity Patterns ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Neural Net ◽  
Dependent Manner ◽  
Tracking Data ◽  
Average Accuracy

Augmented reality is the fusion of virtual components and our real surroundings. The simultaneous visibility of generated and natural objects often requires users to direct their selective attention to a specific target that is either real or virtual. In this study, we investigated whether this target is real or virtual by using machine learning techniques to classify electroencephalographic (EEG) and eye tracking data collected in augmented reality scenarios. A shallow convolutional neural net classified 3 second EEG data windows from 20 participants in a person-dependent manner with an average accuracy above 70% if the testing data and training data came from different trials. This accuracy could be significantly increased to 77% using a multimodal late fusion approach that included the recorded eye tracking data. Person-independent EEG classification was possible above chance level for 6 out of 20 participants. Thus, the reliability of such a brain–computer interface is high enough for it to be treated as a useful input mechanism for augmented reality applications.

scholarly journals Imaging Time Series of Eye Tracking Data to Classify Attentional States

2021 ◽  
Vol 15 ◽  
Author(s):  
Lisa-Marie Vortmann ◽  
Jannes Knychalla ◽  
Sonja Annerer-Walcher ◽  
Mathias Benedek ◽  
Felix Putze
Keyword(s):  
Time Series ◽  
Eye Tracking ◽  
Eye Gaze ◽  
Image Features ◽  
Data Sets ◽  
Neural Net ◽  
Tracking Data ◽  
Attentional State ◽  
Gaze Behavior ◽  
Imaging Time

It has been shown that conclusions about the human mental state can be drawn from eye gaze behavior by several previous studies. For this reason, eye tracking recordings are suitable as input data for attentional state classifiers. In current state-of-the-art studies, the extracted eye tracking feature set usually consists of descriptive statistics about specific eye movement characteristics (i.e., fixations, saccades, blinks, vergence, and pupil dilation). We suggest an Imaging Time Series approach for eye tracking data followed by classification using a convolutional neural net to improve the classification accuracy. We compared multiple algorithms that used the one-dimensional statistical summary feature set as input with two different implementations of the newly suggested method for three different data sets that target different aspects of attention. The results show that our two-dimensional image features with the convolutional neural net outperform the classical classifiers for most analyses, especially regarding generalization over participants and tasks. We conclude that current attentional state classifiers that are based on eye tracking can be optimized by adjusting the feature set while requiring less feature engineering and our future work will focus on a more detailed and suited investigation of this approach for other scenarios and data sets.

scholarly journals Decoding activity in Broca's area predicts the occurrence of auditory hallucinations across subjects

2021 ◽  
Author(s):  
Thomas Fovet ◽  
Pierre Yger ◽  
Renaud Lopes ◽  
Amicie de Pierrefeu ◽  
Edouard Duchesnay ◽  
...  
Keyword(s):  
Brain Activity ◽  
Activity Patterns ◽  
Computational Cost ◽  
Control Sample ◽  
Area Under The Curve ◽  
Machine Learning Techniques ◽  
Imagery Task ◽  
Broca’S Area ◽  
Broca's Area ◽  
Auditory Verbal Hallucinations

BACKGROUND: Functional magnetic resonance imaging (fMRI) capture aims at detecting auditory-verbal hallucinations (AVHs) from continuously recorded brain activity. Establishing efficient capture methods with low computational cost that easily generalize between patients remains a key objective in precision psychiatry. To address this issue, we developed a novel automatized fMRI-capture procedure for AVHs in schizophrenia patients. METHODS: We used a previously validated, but labor-intensive, personalized fMRI-capture method to train a linear classifier using machine-learning techniques. We benchmarked the performances of this classifier on 2320 AVH periods vs. resting-state periods obtained from schizophrenia patients with frequent symptoms (n=23). We characterized patterns of BOLD activity that were predictive of AVH both within- and between-subjects. Generalizability was assessed with a second independent sample gathering 2000 AVH labels (n=34 schizophrenia patients), while specificity was tested with a nonclinical control sample performing an auditory imagery task (840 labels, n=20). RESULTS: Our between-subject classifier achieved high decoding accuracy (area-under-the-curve, AUC = 0.85) and discriminated AVH from rest and verbal imagery. Optimizing the parameters on the first schizophrenia dataset and testing its performance on the second dataset led to a 0.85 out-of-sample AUC (0.88 for the converse test). We showed that AVH detection critically depends on local BOLD activity patterns within Broca's area. CONCLUSIONS: Our results demonstrate that it is possible to reliably detect AVH-states from BOLD signals in schizophrenia patients using a multivariate decoder without performing complex regularization procedures. These findings constitute a crucial step toward brain-based treatments for severe drug-resistant hallucinations.

scholarly journals Inter-individual deep image reconstruction

2022 ◽  
Author(s):  
Jun Kai Ho ◽  
Tomoyasu Horikawa ◽  
Kei Majima ◽  
Yukiyasu Kamitani
Keyword(s):  
Image Reconstruction ◽  
Brain Activity ◽  
Activity Patterns ◽  
Visual Image ◽  
Training Data ◽  
Visual Features ◽  
Fine Grained ◽  
Visual Areas ◽  
Brain Activity Pattern ◽  
Visual Cortical

The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. While anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual contents. In this study, we evaluated machine learning models called neural code converters that predict one's brain activity pattern (target) from another's (source) given the same stimulus by the decoding of hierarchical visual features and the reconstruction of perceived images. The training data for converters consisted of fMRI data obtained with identical sets of natural images presented to pairs of individuals. Converters were trained using the whole visual cortical voxels from V1 through the ventral object areas, without explicit labels of visual areas. We decoded the converted brain activity patterns into hierarchical visual features of a deep neural network (DNN) using decoders pre-trained on the target brain and then reconstructed images via the decoded features. Without explicit information about visual cortical hierarchy, the converters automatically learned the correspondence between the visual areas of the same levels. DNN feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The viewed images were faithfully reconstructed with recognizable silhouettes of objects even with relatively small amounts of data for converter training. The conversion also allows pooling data across multiple individuals, leading to stably high reconstruction accuracy compared to those converted between individuals. These results demonstrate that the conversion learns hierarchical correspondence and preserves the fine-grained representations of visual features, enabling visual image reconstruction using decoders trained on other individuals.

Automatic Analysis of Eye-Tracking Data for Augmented Reality Applications: A Prospective Outlook

2016 ◽  
pp. 217-230 ◽  
Author(s):  
Simona Naspetti ◽  
Roberto Pierdicca ◽  
Serena Mandolesi ◽  
Marina Paolanti ◽  
Emanuele Frontoni ◽  
...  
Keyword(s):  
Augmented Reality ◽  
Eye Tracking ◽  
Automatic Analysis ◽  
Tracking Data

scholarly journals MR-based camera-less eye tracking using deep neural networks

2020 ◽  
Author(s):  
Markus Frey ◽  
Matthias Nau ◽  
Christian F. Doeller
Keyword(s):  
Eye Tracking ◽  
Open Source Software ◽  
Deep Neural Networks ◽  
Brain Activity ◽  
Important Variable ◽  
Training Data ◽  
Human Cognition ◽  
Clinical Settings ◽  
Oculomotor Function ◽  
Gaze Position

AbstractViewing behavior provides a window into many central aspects of human cognition and health, and it is an important variable of interest or confound in many fMRI studies. To make eye tracking freely and widely available for MRI research, we developed DeepMReye: a convolutional neural network that decodes gaze position from the MR-signal of the eyeballs. It performs camera-less eye tracking at sub-imaging temporal resolution in held-out participants with little training data and across a broad range of scanning protocols. Critically, it works even in existing datasets and when the eyes are closed. Decoded eye movements explain network-wide brain activity also in regions not associated with oculomotor function. This work emphasizes the importance of eye tracking for the interpretation of fMRI results and provides an open-source software solution that is widely applicable in research and clinical settings.

scholarly journals Affective computing with eye-tracking data in the study of the visual perception of architectural spaces

2019 ◽  
Vol 252 ◽  
pp. 03021
Author(s):  
Magdalena Chmielewska ◽  
Mariusz Dzieńkowski ◽  
Jacek Bogucki ◽  
Wojciech Kocki ◽  
Bartłomiej Kwiatkowski ◽  
...  
Keyword(s):  
Eye Tracking ◽  
Affective Computing ◽  
Selection Process ◽  
Binary Classification ◽  
Activity Patterns ◽  
Pupillary Response ◽  
Accuracy Score ◽  
Tracking Data ◽  
Eye Tracker ◽  
Architectural Spaces

In the presented study the usefulness of eye-tracking data for classification of architectural spaces as stressful or relaxing was examined. The eye movements and pupillary response data were collected using the eye-tracker from 202 adult volunteers in the laboratory experiment in a well-controlled environment. Twenty features were extracted from the eye-tracking data and after the selection process the features were used in automated binary classification with a variety of machine learning classifiers including neural networks. The results of the classification using eye-tracking data features yielded 68% accuracy score, which can be considered satisfactory. Moreover, statistical analysis showed statistically significant differences in eye activity patterns between visualisations labelled as stressful or relaxing.

scholarly journals Functional magnetic resonance imaging-based brain decoding with visual semantic model

2020 ◽  
Vol 10 (6) ◽  
pp. 6682
Author(s):  
Piyawat Saengpetch ◽  
Luepol Pipanmemekaporn ◽  
Suwatchai Kamolsantiroj
Keyword(s):  
Magnetic Resonance Imaging ◽  
Magnetic Resonance ◽  
Brain Activity ◽  
Activity Patterns ◽  
Experimental Studies ◽  
Training Data ◽  
Semantic Features ◽  
Resonance Imaging ◽  
Brain Decoding ◽  
The Brain

The activity pattern of the brain has been activated to identify a person in mind. Using the function magnetic resonance imaging (fMRI) to decipher brain decoding is the most accepted method. However, the accuracy of fMRI-based brain decoder is still restricted due to limited training samples. The limitations of the brain decoder using fMRI are passed through the design features proposed for many label coding and model training to predict these characteristics for a particular label. Moreover, what kind of semantic features for deciphering the neurological activity patterns are unclear. In current work, a new calculation model for learning decoding labels that is consistent with fMRI activity responses. The approach demonstrates the proposed corresponding label's success in terms of accuracy, which is decoded from brain activity patterns and compared with conventional text-derived feature technique. Besides, experimental studies present a training model based on multi-tasking to reduce the problems of limited training data sets. Therefore, the multi-task learning model is more efficient than modern methods of calculation, and decoding features may be easily obtained.

The Opposition of Surprisal and Semantic Similarity in the Prediction of Language Processing: Evidence from Eye-tracking Data

2020 ◽  
Author(s):  
Kun Sun
Keyword(s):  
Eye Tracking ◽  
Semantic Similarity ◽  
Cognitive Processing ◽  
Language Processing ◽  
Language Comprehension ◽  
Word Processing ◽  
Reading Time ◽  
Computational Models ◽  
Tracking Data ◽  
Dynamic Approach

Expectations or predictions about upcoming content play an important role during language comprehension and processing. One important aspect of recent studies of language comprehension and processing concerns the estimation of the upcoming words in a sentence or discourse. Many studies have used eye-tracking data to explore computational and cognitive models for contextual word predictions and word processing. Eye-tracking data has previously been widely explored with a view to investigating the factors that influence word prediction. However, these studies are problematic on several levels, including the stimuli, corpora, statistical tools they applied. Although various computational models have been proposed for simulating contextual word predictions, past studies usually preferred to use a single computational model. The disadvantage of this is that it often cannot give an adequate account of cognitive processing in language comprehension. To avoid these problems, this study draws upon a massive natural and coherent discourse as stimuli in collecting the data on reading time. This study trains two state-of-art computational models (surprisal and semantic (dis)similarity from word vectors by linear discriminative learning (LDL)), measuring knowledge of both the syntagmatic and paradigmatic structure of language. We develop a `dynamic approach' to compute semantic (dis)similarity. It is the first time that these two computational models have been merged. Models are evaluated using advanced statistical methods. Meanwhile, in order to test the efficiency of our approach, one recently developed cosine method of computing semantic (dis)similarity based on word vectors data adopted is used to compare with our `dynamic' approach. The two computational and fixed-effect statistical models can be used to cross-verify the findings, thus ensuring that the result is reliable. All results support that surprisal and semantic similarity are opposed in the prediction of the reading time of words although both can make good predictions. Additionally, our `dynamic' approach performs better than the popular cosine method. The findings of this study are therefore of significance with regard to acquiring a better understanding how humans process words in a real-world context and how they make predictions in language cognition and processing.

Sign in / Sign up

Export Citation Format

Share Document