scholarly journals Simultaneous fMRI and eye gaze recordings during prolonged natural stimulation - a studyforrest extension

2016 ◽  
Author(s):  
Michael Hanke ◽  
Nico Adelhöfer ◽  
Daniel Kottke ◽  
Vittorio Iacovella ◽  
Ayan Sengupta ◽  
...  
Keyword(s):  
Language Processing ◽  
Eye Gaze ◽  
Fmri Data ◽  
Stimulus Motion ◽  
Natural Stimulus ◽  
Attentional Processes ◽  
Natural Stimulation ◽  
Potential Impact ◽  
Response Congruency ◽  
Additional Sample

AbstractHere we present an update of the studyforrest (http://studyforrest.org) dataset that complements the previously released functional magnetic resonance imaging (fMRI) data for natural language processing with a new two-hour 3Tesla fMRI acquisition while 15 of the original participants were shown an audio-visual version of the stimulus motion picture. We demonstrate with two validation analyses that these new data support modeling specific properties of the complex natural stimulus, as well as a substantial within-subject BOLD response congruency in brain areas related to the processing of auditory inputs, speech, and narrative when compared to the existing fMRI data for audio-only stimulation. In addition, we provide participants’ eye gaze location as recorded simultaneously with fMRI, and an additional sample of 15 control participants whose eye gaze trajectories for the entire movie were recorded in a lab setting — to enable studies on attentional processes and comparative investigations on the potential impact of the stimulation setting on these processes.

scholarly journals When brain damage “improves” perception: neglect patients can localize motion-shifted probes better than controls

2015 ◽  
Vol 114 (6) ◽  
pp. 3351-3358 ◽  
Author(s):  
Stefania de Vito ◽  
Marine Lunven ◽  
Clémence Bourlon ◽  
Christophe Duret ◽  
Patrick Cavanagh ◽  
...  
Keyword(s):  
Right Hemisphere ◽  
Visual Fields ◽  
Critical Factor ◽  
Significant Shift ◽  
Stimulus Motion ◽  
Attentional Processes ◽  
Position Shift ◽  
The Right ◽  
High Level ◽  
Better Than

When we look at bars flashed against a moving background, we see them displaced in the direction of the upcoming motion (flash-grab illusion). It is still debated whether these motion-induced position shifts are low-level, reflexive consequences of stimulus motion or high-level compensation engaged only when the stimulus is tracked with attention. To investigate whether attention is a causal factor for this striking illusory position shift, we evaluated the flash-grab illusion in six patients with damaged attentional networks in the right hemisphere and signs of left visual neglect and six age-matched controls. With stimuli in the top, right, and bottom visual fields, neglect patients experienced the same amount of illusion as controls. However, patients showed no significant shift when the test was presented in their left hemifield, despite having equally precise judgments. Thus, paradoxically, neglect patients perceived the position of the flash more veridically in their neglected hemifield. These results suggest that impaired attentional processes can reduce the interaction between a moving background and a superimposed stationary flash, and indicate that attention is a critical factor in generating the illusory motion-induced shifts of location.

Multi-modal referring expressions in human-human task descriptions and their implications for human-robot interaction

2016 ◽  
Vol 17 (2) ◽  
pp. 180-210
Author(s):  
Stephanie Gross ◽  
Brigitte Krenn ◽  
Matthias Scheutz
Keyword(s):  
Language Processing ◽  
Eye Gaze ◽  
Human Robot Interaction ◽  
Referring Expressions ◽  
Visual Channel ◽  
Student Interactions ◽  
Seamless Integration ◽  
Teacher Student ◽  
Processing Module ◽  
Task Descriptions

Abstract Human instructors often refer to objects and actions involved in a task description using both linguistic and non-linguistic means of communication. Hence, for robots to engage in natural human-robot interactions, we need to better understand the various relevant aspects of human multi-modal task descriptions. We analyse reference resolution to objects in a data collection comprising two object manipulation tasks (22 teacher student interactions in Task 1 and 16 in Task 2) and find that 78.76% of all referring expressions to the objects relevant in Task 1 are verbally underspecified and 88.64% of all referring expressions are verbally underspecified in Task 2. The data strongly suggests that a language processing module for robots must be genuinely multi-modal, allowing for seamless integration of information transmitted in the verbal and the visual channel, whereby tracking the speaker’s eye gaze and gestures as well as object recognition are necessary preconditions.

scholarly journals Beyond robotic speech: mutual benefits to cognitive psychology and artificial intelligence from the study of multimodal communication

2020 ◽  
Author(s):  
Beata Grzyb ◽  
Gabriella Vigliocco
Keyword(s):  
Artificial Intelligence ◽  
Cognitive Psychology ◽  
Language Learning ◽  
Language Processing ◽  
Computational Models ◽  
Eye Gaze ◽  
Human Communication ◽  
Embodied Agents ◽  
Human Machine Interaction ◽  
Neural Underpinnings

Language has predominately been studied as a unimodal phenomenon - as speech or text without much consideration of its physical and social context – this is true both in cognitive psychology/psycholinguistics as well as in artificial intelligence. However, in everyday life, language is most often used in face-to-face communication and in addition to structured speech it comprises a dynamic system of multiplex components such as gestures, eye gaze, mouth movements and prosodic modulation. Recently, cognitive scientists have started to realise the potential importance of multimodality for the understanding of human communication and its neural underpinnings; while AI scientists have begun to address how to integrate multimodality in order to improve communication between human and artificial embodied agent. We review here the existing literature on multimodal language learning and processing in humans and the literature on perception of artificial agents, their comprehension and production of multimodal cues and we discuss their main limitations. We conclude by arguing that by joining forces AI scientists can improve the effectiveness of human-machine interaction and increase the human-likeness and acceptance of embodied agents in society. In turn, computational models that generate language in artificial embodied agents constitute a unique research tool to investigate the underlying mechanisms that govern language processing and learning in humans.

Attention in Early Development

2021 ◽  
Author(s):  
Stefania Conte ◽  
John Richards
Keyword(s):  
Eye Gaze ◽  
Brain Network ◽  
Intervention Programs ◽  
Typical Development ◽  
Physiological Measures ◽  
Attentional Processes ◽  
Cognitive Domains ◽  
Functional Development ◽  
Guide Intervention ◽  
Neuroimaging Techniques

Attention is a complex construct that shows development throughout the life span and undergoes significant changes over the first years of life. The complexity of attentional processes is described by the different systems and brain network theorized to describe the construct (i.e., alerting, orienting, executive attention, and sustained attention). Evidence of the development of attention in infancy comes from several behavioral paradigms—primarily focused on the analysis of infants’ eye gaze—physiological measures, and neuroimaging techniques. Many of the changes in attention rely upon the structural and functional development of brain areas involved in attention processes. Behavioral and physiological signs mark the development of attention and are identifiable very early in life. The investigation of the typical development of attention is pivotal for the understanding of atypical trajectories that characterize many neurodevelopmental disorders. The individuation of alterations in early visual attention processes may be utilized to guide intervention programs aimed at improving attention and other cognitive domains.

scholarly journals Investigating age differences in the influence of joint attention on working memory: a preregistered online study.

2021 ◽  
Author(s):  
Samantha Gregory ◽  
Klaus Kessler
Keyword(s):  
Working Memory ◽  
Eye Gaze ◽  
Younger Adults ◽  
Attentional Processes ◽  
Online Study ◽  
Age Related ◽  
Dynamic Head ◽  
Gaze Cuing ◽  
Important Evidence ◽  
Gaze Cues

Previous research has demonstrated that older adults make limited use of social cues as compared to younger adults. This has been investigated by testing the influence of gaze cues on attentional processes, with findings showing significantly smaller gaze cuing effects for older than younger adults. Here we aimed to investigate whether this would also result in age related differences in the influence of gaze cues on working memory. We therefore tested the effects of gaze cues from realistic human avatars on working memory across two experiments using dynamic head turns and more subtle eye gaze movements. Results demonstrated that for both older and younger adults, gaze cues influenced working memory processes, though there were some important differences related to the nature of the cue. Overall, we provide important evidence that sharing attention benefits cognition across the lifespan.

scholarly journals The Contributions of Eye Gaze Fixations and Target-Lure Similarity to Behavioral and fMRI Indices of Pattern Separation and Pattern Completion

2021 ◽  
Author(s):  
Daniel K Bjornn ◽  
Julie Van ◽  
Brock Kirwan
Keyword(s):  
Eye Movements ◽  
False Alarm ◽  
Eye Movement ◽  
Eye Gaze ◽  
Recognition Task ◽  
Discrimination Performance ◽  
Pattern Separation ◽  
Fmri Data ◽  
Pattern Completion ◽  
Gaze Fixations

Pattern separation and pattern completion are generally studied in humans using mnemonic discrimination tasks such as the Mnemonic Similarity Task (MST) where participants identify similar lures and repeated items from a series of images. Failures to correctly discriminate lures are thought to reflect a failure of pattern separation and a propensity toward pattern completion. Recent research has challenged this perspective, suggesting that poor encoding rather than pattern completion accounts for the occurrence of false alarm responses to similar lures. In two experiments, participants completed a continuous recognition task version of the MST while eye movement (Experiment 1 and 2) and fMRI data (Experiment 2) were collected. While we replicated the result that fixation counts at study predicted accuracy on lure trials, we found that target-lure similarity was a much stronger predictor of accuracy on lure trials across both experiments. Lastly, we found that fMRI activation changes in the hippocampus were significantly correlated with the number of fixations at study for correct but not incorrect mnemonic discrimination judgments when controlling for target-lure similarity. Our findings indicate that while eye movements during encoding predict subsequent hippocampal activation changes, mnemonic discrimination performance is better described by pattern separation and pattern completion processes that are influenced by target-lure similarity than simply poor encoding.

scholarly journals Presurgical Language fMRI and Postsurgical Deficits: A Single Centre Experience

2013 ◽  
Vol 40 (6) ◽  
pp. 819-823
Author(s):  
Charmaine Wiens ◽  
Bradley G. Goodyear ◽  
Ankur Goel ◽  
Paolo Federico ◽  
Walter Hader ◽  
...  
Keyword(s):  
Language Processing ◽  
Surgical Planning ◽  
Single Centre ◽  
Language Deficits ◽  
Language Lateralization ◽  
Post Surgery ◽  
Single Centre Experience ◽  
Clinical Fmri ◽  
Hemisphere Dominance ◽  
Potential Impact

Background:In this study, we conducted a retrospective investigation of our initial single-centre experience with the clinical use of functional magnetic resonance imaging (fMRI) of hemisphere dominance for language processing (i.e., language lateralization). We demonstrated its association with surgical outcome and its potential impact on surgical planning and patient management.Methods:Fifty-two cases were reviewed, covering the period from July 2007 to July 2010. Clinical fMRI reports were examined to determine the hemisphere dominance for language processing. Neurological reports were examined to determine if new language deficits were present post-surgery. Neurosurgeon notes were also reviewed to determine if fMRI had an impact on surgical planning.Results:Of the cases reviewed, 49 (94%) generated conclusive fMRI. Eleven (22%) patients exhibited fMRI language lateralization contralateral to pathology; zero of nine of these patients that had surgery experienced post-surgical deficits. Twenty-two (44%) patients exhibited fMRI language lateralization ipsilateral to pathology; three of 13 of these patients that had surgery experienced post-surgical deficits. Sixteen (34%) patients exhibited bilateral lateralization of language; five of 13 of these patients that had surgery experienced post-surgery deficits. Several post-fMRI reports indicated that fMRI results had an impact on surgical planning.Conclusions:Our results suggest that fMRI demonstrations of language processing within the hemisphere ipsilateral to pathology (either ipsilateral alone or bilateral) is associated with a greater risk for post-surgical language deficits, and in these cases, fMRI results should be taken into consideration for pre-surgical planning.

scholarly journals A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation

2016 ◽  
Vol 3 (1) ◽  
Author(s):  
Michael Hanke ◽  
Nico Adelhöfer ◽  
Daniel Kottke ◽  
Vittorio Iacovella ◽  
Ayan Sengupta ◽  
...  
Keyword(s):  
Eye Gaze ◽  
Natural Stimulation

The Use of Eye-Tracking in Spatial Thinking Research

2019 ◽  
pp. 588-609
Author(s):  
Alina Nazareth ◽  
Rosalie Odean ◽  
Shannon M. Pruden
Keyword(s):  
Eye Tracking ◽  
Language Processing ◽  
Cognitive Processes ◽  
Response Times ◽  
Eye Gaze ◽  
Cognitive Strategy ◽  
Spatial Thinking ◽  
Educational Pedagogy ◽  
Spatial Tasks ◽  
Tracking Technology

This chapter highlights the benefits of eye-tracking technology in spatial thinking research, specifically in the study of complex cognitive processes used to solve spatial tasks including cognitive strategy selection, cognitive strategy flexibility and spatial language processing. The consistent sex differences found in spatial thinking research (i.e., mental rotation), with males outperforming females, is concerning given the link between spatial ability and success in the STEM fields. Traditional methods like self-reports, checklists and response times methods may not be sufficient to study complex cognitive processes. Advances in eye-tracking technology make it possible to efficiently record and analyze voluminous eye-gaze data as an indirect measure of underlying cognitive processes involved in solving spatial tasks. A better understanding of the cognitive processes underlying spatial thinking will facilitate the design of effective training and educational pedagogy that encourages spatial thinking across both males and females.

scholarly journals Assessing the effects of cocaine dependence and pathological gambling using group-wise sparse representation of natural stimulus FMRI data

2016 ◽  
Vol 11 (4) ◽  
pp. 1179-1191 ◽  
Author(s):  
Yudan Ren ◽  
Jun Fang ◽  
Jinglei Lv ◽  
Xintao Hu ◽  
Cong Christine Guo ◽  
...  
Keyword(s):  
Pathological Gambling ◽  
Sparse Representation ◽  
Cocaine Dependence ◽  
Fmri Data ◽  
Natural Stimulus
Sign in / Sign up

Export Citation Format

Share Document