Human brain activity during mental imagery exhibits signatures of inference in a hierarchical generative model

2018 ◽  
Author(s):  
Jesse Breedlove ◽  
Ghislain St-Yves ◽  
Cheryl Olman ◽  
Thomas Naselaris
Keyword(s):  
Mental Imagery ◽  
Human Subjects ◽  
Brain Activity ◽  
Generative Model ◽  
Formal Structure ◽  
Low Level ◽  
Visual Encoding ◽  
Visual Areas ◽  
High Level ◽  
Echo Effect

Humans have long wondered about the function of mental imagery and its relationship to vision. Although visual representations are utilized during imagery, the computations they subserve are unclear. Building on a theory that treats vision as inference about the causes of sensory stimulation in an internal generative model, we propose that mental imagery is inference about the sensory consequences of predicted or remembered causes. The relation between these complementary inferences yields a relation between the brain activity patterns associated with imagery and vision. We show that this relation has the formal structure of an echo that makes encoding of imagined stimuli in low-level visual areas resemble the encoding of seen stimuli in higher areas. To test for evidence of this echo effect we developed imagery encoding models—a new tool for revealing how imagined stimuli are encoded in brain activity. We estimated imagery encoding models from brain activity measured with fMRI while human subjects imagined complex visual stimuli, and then compared these to visual encoding models estimated from a matched viewing experiment. Consistent with an echo effect, imagery encoding models in low-level visual areas exhibited decreased spatial frequency preference and larger, more foveal receptive fields, thus resembling visual encoding models in high-level visual areas where imagery and vision appeared to be almost interchangeable. Our findings support an interpretation of mental imagery as a predictive inference that is conditioned on activity in high-level visual cortex, and is related to vision through shared dependence on an internal model of the visual world.

scholarly journals FFA and OFA encode distinct types of face identity information

2020 ◽  
Author(s):  
Maria Tsantani ◽  
Nikolaus Kriegeskorte ◽  
Katherine Storrs ◽  
Adrian Lloyd Williams ◽  
Carolyn McGettigan ◽  
...  
Keyword(s):  
Brain Activity ◽  
Activity Patterns ◽  
Brain Regions ◽  
Image Encoding ◽  
Low Level ◽  
Social Traits ◽  
Face Area ◽  
Face Similarity ◽  
And Gender ◽  
High Level

AbstractFaces of different people elicit distinct functional MRI (fMRI) patterns in several face-selective brain regions. Here we used representational similarity analysis to investigate what type of identity-distinguishing information is encoded in three face-selective regions: fusiform face area (FFA), occipital face area (OFA), and posterior superior temporal sulcus (pSTS). We used fMRI to measure brain activity patterns elicited by naturalistic videos of famous face identities, and compared their representational distances in each region with models of the differences between identities. Models included low-level to high-level image-computable properties and complex human-rated properties. We found that the FFA representation reflected perceived face similarity, social traits, and gender, and was well accounted for by the OpenFace model (deep neural network, trained to cluster faces by identity). The OFA encoded low-level image-based properties (pixel-wise and Gabor-jet dissimilarities). Our results suggest that, although FFA and OFA can both discriminate between identities, the FFA representation is further removed from the image, encoding higher-level perceptual and social face information.

scholarly journals Representation of real-world event schemas during narrative perception

2018 ◽  
Author(s):  
Christopher Baldassano ◽  
Uri Hasson ◽  
Kenneth A. Norman
Keyword(s):  
Human Subjects ◽  
Brain Activity ◽  
Temporal Structure ◽  
Animal Experiments ◽  
Situation Model ◽  
Brain Regions ◽  
Control Group ◽  
Daily Lives ◽  
High Level ◽  
Almost All

AbstractUnderstanding movies and stories requires maintaining a high-level situation model that abstracts away from perceptual details to describe the location, characters, actions, and causal relationships of the currently unfolding event. These models are built not only from information present in the current narrative, but also from prior knowledge about schematic event scripts, which describe typical event sequences encountered throughout a lifetime. We analyzed fMRI data from 44 human subjects presented with sixteen three-minute stories, consisting of four schematic events drawn from two different scripts (eating at a restaurant or going through the airport). Aside from this shared script structure, the stories varied widely in terms of their characters and storylines, and were presented in two highly dissimilar formats (audiovisual clips or spoken narration). One group was presented with the stories in an intact temporal sequence, while a separate control group was presented with the same events in scrambled order. Regions including the posterior medial cortex, medial prefrontal cortex (mPFC), and superior frontal gyrus exhibited schematic event patterns that generalized across stories, subjects, and modalities. Patterns in mPFC were also sensitive to overall script structure, with temporally scrambled events evoking weaker schematic representations. Using a Hidden Markov Model, patterns in these regions can predict the script (restaurant vs. airport) of unlabeled data with high accuracy, and can be used to temporally align multiple stories with a shared script. These results extend work on the perception of controlled, artificial schemas in human and animal experiments to naturalistic perception of complex narrative stimuli.Significance StatementIn almost all situations we encounter in our daily lives, we are able to draw on our schematic knowledge about what typically happens in the world to better perceive and mentally represent our ongoing experiences. In contrast to previous studies that investigated schematic cognition using simple, artificial associations, we measured brain activity from subjects watching movies and listening to stories depicting restaurant or airport experiences. Our results reveal a network of brain regions that is sensitive to the shared temporal structure of these naturalistic situations. These regions abstract away from the particular details of each story, activating a representation of the general type of situation being perceived.

scholarly journals Alterations of Regional Spontaneous Brain Activity and Gray Matter Volume in the Blind

2015 ◽  
Vol 2015 ◽  
pp. 1-12 ◽  
Author(s):  
Aili Jiang ◽  
Jing Tian ◽  
Rui Li ◽  
Yong Liu ◽  
Tianzi Jiang ◽  
...  
Keyword(s):  
Gray Matter ◽  
Brain Activity ◽  
Gray Matter Volume ◽  
Brain Regions ◽  
Visual Deprivation ◽  
Onset Age ◽  
Spontaneous Brain Activity ◽  
Matter Volume ◽  
Visual Areas ◽  
High Level

Visual deprivation can induce alterations of regional spontaneous brain activity (RSBA). However, the effects of onset age of blindness on the RSBA and the association between the alterations of RSBA and brain structure are still unclear in the blind. In this study, we performed resting-state functional and structural magnetic resonance imaging on 50 sighted controls and 91 blind subjects (20 congenitally blind, 27 early blind, and 44 late blind individuals). Compared with the sighted control, we identified increased RSBA in the blind in primary and high-level visual areas and decreased RSBA in brain regions which are ascribed to sensorimotor and salience networks. In contrast, blind subjects exhibited significantly decreased gray matter volume (GMV) in the visual areas, while they exhibited significantly increased GMV in the sensorimotor areas. Moreover, the onset age of blindness was negatively correlated with the GMV of visual areas in blind subjects, whereas it exerted complex influences on the RSBA. Finally, significant negative correlations were shown between RSBA and GMV values. Our results demonstrated system-dependent, inverse alterations in RSBA and GMV after visual deprivation. Furthermore, the onset age of blindness has different effects on the reorganizations in RSBA and GMV.

scholarly journals The rhythms of predictive coding: pre-stimulus phase modulates the influence of shape perception on luminance judgments

2016 ◽  
Author(s):  
Biao Han ◽  
Rufin VanRullen
Keyword(s):  
Temporal Dynamics ◽  
Human Subjects ◽  
Predictive Coding ◽  
Shape Perception ◽  
Low Level ◽  
3D Shape ◽  
Coding Efficiency ◽  
Spatial Response ◽  
Left And Right ◽  
High Level

AbstractPredictive coding is an influential model emphasizing interactions between feedforward and feedback signals. Here, we investigated its temporal dynamics. Two gray disks with different versions of the same stimulus, one enabling predictive feedback (a 3D-shape) and one impeding it (random-lines), were simultaneously presented on the left and right of fixation. Human subjects judged the luminance of the two disks while EEG was recorded. Independently of the spatial response (left/right), we found that the choice of 3D-shape or random-lines as the brighter disk (our measure of post-stimulus predictive coding efficiency on each trial) fluctuated along with the pre-stimulus phase of two spontaneous oscillations: a ~5Hz oscillation in contralateral frontal electrodes and a ~16Hz oscillation in contralateral occipital electrodes. This pattern of results demonstrates that predictive coding is a rhythmic process, and suggests that it could take advantage of faster oscillations in low-level areas and slower oscillations in high-level areas.

scholarly journals Inferior parietal lobule and early visual areas support elicitation of individualized meanings during narrative listening

2018 ◽  
Author(s):  
S. Saalasti ◽  
J. Alho ◽  
M. Bar ◽  
E. Glerean ◽  
T. Honkela ◽  
...  
Keyword(s):  
Individual Differences ◽  
Mental Imagery ◽  
Latent Semantic Analysis ◽  
Semantic Analysis ◽  
Brain Activity ◽  
Brain Regions ◽  
Inferior Parietal Lobule ◽  
Semantic Representations ◽  
Visual Areas ◽  
Parietal Lobule

AbstractWhen listening to a narrative, the verbal expressions translate into meanings and flow of mental imagery, at best vividly immersing the keen listener into the sights, sounds, scents, objects, actions, and events in the story. However, the same narrative can be heard quite differently based on differences in listeners’ previous experiences and knowledge, as the semantics and mental imagery elicited by words and phrases in the story vary extensively between any given two individuals. Here, we capitalized on such inter-individual differences to disclose brain regions that support transformation of narrative into individualized propositional meanings and associated mental imagery by analyzing brain activity associated with behaviorally-assessed individual meanings elicited by a narrative. Sixteen subjects listed words best describing what had come to their minds during each 3–5 sec segment of an eight-minute narrative that they listened during fMRI of brain hemodynamic activity. Similarities in these word listings between subjects, estimated using latent-semantic analysis combined with WordNet knowledge, predicted similarities in brain hemodynamic activity in supramarginal and angular gyri as well as in cuneus. Our results demonstrate how inter-individual differences in semantic representations can be measured and utilized to identify specific brain regions that support the elicitation of individual propositional meanings and the associated mental imagery when one listens to a narrative.

scholarly journals Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder

2021 ◽  
Vol 17 (3) ◽  
pp. e1008775
Author(s):  
Haider Al-Tahan ◽  
Yalda Mohsenzadeh
Keyword(s):  
Visual Information ◽  
Activity Patterns ◽  
Visual Pathway ◽  
Generative Model ◽  
Data Set ◽  
Low Level ◽  
Ventral Visual Pathway ◽  
Neural Feedback ◽  
High Level ◽  
Diverse Data

While vision evokes a dense network of feedforward and feedback neural processes in the brain, visual processes are primarily modeled with feedforward hierarchical neural networks, leaving the computational role of feedback processes poorly understood. Here, we developed a generative autoencoder neural network model and adversarially trained it on a categorically diverse data set of images. We hypothesized that the feedback processes in the ventral visual pathway can be represented by reconstruction of the visual information performed by the generative model. We compared representational similarity of the activity patterns in the proposed model with temporal (magnetoencephalography) and spatial (functional magnetic resonance imaging) visual brain responses. The proposed generative model identified two segregated neural dynamics in the visual brain. A temporal hierarchy of processes transforming low level visual information into high level semantics in the feedforward sweep, and a temporally later dynamics of inverse processes reconstructing low level visual information from a high level latent representation in the feedback sweep. Our results append to previous studies on neural feedback processes by presenting a new insight into the algorithmic function and the information carried by the feedback processes in the ventral visual pathway.

scholarly journals Activity in early visual areas predicts interindividual differences in binocular rivalry dynamics

2014 ◽  
Vol 111 (6) ◽  
pp. 1190-1202 ◽  
Author(s):  
Hiroyuki Yamashiro ◽  
Hiroki Yamamoto ◽  
Hiroaki Mano ◽  
Masahiro Umeda ◽  
Toshihiro Higuchi ◽  
...  
Keyword(s):  
Binocular Rivalry ◽  
Brain Activity ◽  
Potential Contribution ◽  
Continuous Flash Suppression ◽  
Visual Areas ◽  
Potential Link ◽  
The Individual ◽  
High Level ◽  
Parietal Cortices ◽  
The Invisible

When dissimilar images are presented to the two eyes, binocular rivalry (BR) occurs, and perception alternates spontaneously between the images. Although neural correlates of the oscillating perception during BR have been found in multiple sites along the visual pathway, the source of BR dynamics is unclear. Psychophysical and modeling studies suggest that both low- and high-level cortical processes underlie BR dynamics. Previous neuroimaging studies have demonstrated the involvement of high-level regions by showing that frontal and parietal cortices responded time locked to spontaneous perceptual alternation in BR. However, a potential contribution of early visual areas to BR dynamics has been overlooked, because these areas also responded to the physical stimulus alternation mimicking BR. In the present study, instead of focusing on activity during perceptual switches, we highlighted brain activity during suppression periods to investigate a potential link between activity in human early visual areas and BR dynamics. We used a strong interocular suppression paradigm called continuous flash suppression to suppress and fluctuate the visibility of a probe stimulus and measured retinotopic responses to the onset of the invisible probe using functional MRI. There were ∼130-fold differences in the median suppression durations across 12 subjects. The individual differences in suppression durations could be predicted by the amplitudes of the retinotopic activity in extrastriate visual areas (V3 and V4v) evoked by the invisible probe. Weaker responses were associated with longer suppression durations. These results demonstrate that retinotopic representations in early visual areas play a role in the dynamics of perceptual alternations during BR.

scholarly journals Neuroimaging Findings on Amodal Completion: A Review

i-Perception ◽  
2019 ◽  
Vol 10 (2) ◽  
pp. 204166951984004 ◽  
Author(s):  
Jordy Thielen ◽  
Sander E. Bosch ◽  
Tessa M. van Leeuwen ◽  
Marcel A. J. van Gerven ◽  
Rob van Lier
Keyword(s):  
Computational Models ◽  
Amodal Completion ◽  
Lateral Occipital Complex ◽  
Low Level ◽  
Face Area ◽  
Perceptual Completion ◽  
Visual Areas ◽  
High Level ◽  
Partially Occluded ◽  
Retinal Information

Amodal completion is the phenomenon of perceiving completed objects even though physically they are partially occluded. In this review, we provide an extensive overview of the results obtained from a variety of neuroimaging studies on the neural correlates of amodal completion. We discuss whether low-level and high-level cortical areas are implicated in amodal completion; provide an overview of how amodal completion unfolds over time while dissociating feedforward, recurrent, and feedback processes; and discuss how amodal completion is represented at the neuronal level. The involvement of low-level visual areas such as V1 and V2 is not yet clear, while several high-level structures such as the lateral occipital complex and fusiform face area seem invariant to occlusion of objects and faces, respectively, and several motor areas seem to code for object permanence. The variety of results on the timing of amodal completion hints to a mixture of feedforward, recurrent, and feedback processes. We discuss whether the invisible parts of the occluded object are represented as if they were visible, contrary to a high-level representation. While plenty of questions on amodal completion remain, this review presents an overview of the neuroimaging findings reported to date, summarizes several insights from computational models, and connects research of other perceptual completion processes such as modal completion. In all, it is suggested that amodal completion is the solution to deal with various types of incomplete retinal information, and highly depends on stimulus complexity and saliency, and therefore also give rise to a variety of observed neural patterns.

scholarly journals Combinations of low-level and high-level neural processes can account for distinct patterns of context-dependent choice

2018 ◽  
Author(s):  
Mehran Spitmaan ◽  
Oihane Horno ◽  
Emily Chu ◽  
Alireza Soltani
Keyword(s):  
Risk Aversion ◽  
Cognitive Processes ◽  
Context Effects ◽  
Human Subjects ◽  
Risk Preference ◽  
Neural Adaptation ◽  
Low Level ◽  
Neural Representations ◽  
High Level ◽  
Decoy Effects

AbstractContext effects have been explained by either high-level cognitive processes or low-level neural adjustments but not their combination. It is currently unclear how these processes interact to shape individuals’ responses to context. Here, we used a large cohort of human subjects in experiments involving choice between two or three gambles in order to study the dependence of context effects on neural adaptation and individuals’ risk attitudes. We found no evidence that neural adaptation on long timescales (~100 trials) contributes to context effects. However, we identified two groups of subjects with distinct patterns of responses to decoys, both of which depended on individuals’ risk aversion. Subjects in the first group exhibited strong, consistent decoy effects and became more risk averse due to decoy presentation. In contrast, subjects in the second group did not show consistent decoy effects and became more risk seeking. The degree of change in risk aversion due to decoy presentation was positively correlated with the initial degrees of risk aversion. To explain these results and reveal underlying neural mechanisms, we developed a new model that incorporates both low- and high-level processes to fit individuals’ choice behavior. We found that observed decoy effects can be explained by a combination of adjustments in neural representations and competitive weighting of reward attributes, both of which depend on risk aversion but in opposite directions. Altogether, our results demonstrate how a combination of low- and high-level processes shapes multi-attribute choice, modulates overall risk preference, and explains distinct behavioral phenotypes.Significance statementA large body of experimental work has illustrated that the introduction of a new, and often irrelevant, option can influence preference among the existing options, a phenomenon referred to as context or decoy effects. Although context effects have been explained by high-level cognitive processes—such as comparisons and competitions between attributes—or low-level adjustments of neural representations, it is unclear how these processes interact to shape individuals’ responses to context. Here, we show that both high-level cognitive processes and low-level neural adjustments shift risk preference during choice between multiple options but in opposite directions. Moreover, we demonstrate that a combination of these processes can account for distinct patterns of context effects in human subjects.

Sign in / Sign up

Export Citation Format

Share Document