scholarly journals Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis

2016 ◽  
Author(s):  
Jörn Diedrichsen ◽  
Nikolaus Kriegeskorte
Keyword(s):  
Brain Activity ◽  
Activity Patterns ◽  
Similarity Analysis ◽  
Mathematical Framework ◽  
Advantages And Disadvantages ◽  
Representational Similarity Analysis ◽  
Powerful Test ◽  
Activity Measurements ◽  
The Brain ◽  
Representational Models

AbstractRepresentational models specify how activity patterns in populations of neurons (or, more generally, in multivariate brain-activity measurements) relate to sensory stimuli, motor responses, or cognitive processes. In an experimental context, representational models can be defined as hypotheses about the distribution of activity profiles across experimental conditions. Currently, three different methods are being used to test such hypotheses: encoding analysis, pattern component modeling (PCM), and representational similarity analysis (RSA). Here we develop a common mathematical framework for understanding the relationship of these three methods, which share one core commonality: all three evaluate the second moment of the distribution of activity profiles, which determines the representational geometry, and thus how well any feature can be decoded from population activity with any readout mechanism capable of a linear transform. Using simulated data for three different experimental designs, we compare the power of the methods to adjudicate between competing representational models. PCM implements a likelihood-ratio test and therefore provides the most powerful test if its assumptions hold. However, the other two approaches – when conducted appropriately – can perform similarly. In encoding analysis, the linear model needs to be appropriately regularized, which effectively imposes a prior on the activity profiles. With such a prior, an encoding model specifies a well-defined distribution of activity profiles. In RSA, the unequal variances and statistical dependencies of the dissimilarity estimates need to be taken into account to reach near-optimal power in inference. The three methods render different aspects of the information explicit (e.g. single-response tuning in encoding analysis and population-response representational dissimilarity in RSA) and have specific advantages in terms of computational demands, ease of use, and extensibility. The three methods are properly construed as complementary components of a single data-analytical toolkit for understanding neural representations on the basis of multivariate brain-activity data.Author SummaryModern neuroscience can measure activity of many neurons or the local blood oxygenation of many brain locations simultaneously. As the number of simultaneous measurements grows, we can better investigate how the brain represents and transforms information, to enable perception, cognition, and behavior. Recent studies go beyond showing that a brain region is involved in some function. They use representational models that specify how different perceptions, cognitions, and actions are encoded in brain-activity patterns. In this paper, we provide a general mathematical framework for such representational models, which clarifies the relationships between three different methods that are currently used in the neuroscience community. All three methods evaluate the same core feature of the data, but each has distinct advantages and disadvantages. Pattern component modelling (PCM) implements the most powerful test between models, and is analytically tractable and expandable. Representational similarity analysis (RSA) provides a highly useful summary statistic (the dissimilarity) and enables model comparison with weaker distributional assumptions. Finally, encoding models characterize individual responses and enable the study of their layout across cortex. We argue that these methods should be considered components of a larger toolkit for testing hypotheses about the way the brain represents information.

scholarly journals Faces and voices in the brain: a modality-general person-identity representation in superior temporal sulcus

2018 ◽  
Author(s):  
Maria Tsantani ◽  
Nikolaus Kriegeskorte ◽  
Carolyn McGettigan ◽  
Lúcia Garrido
Keyword(s):  
Auditory Processing ◽  
Brain Activity ◽  
Activity Patterns ◽  
Superior Temporal Sulcus ◽  
Brain Regions ◽  
Similarity Analysis ◽  
Representational Similarity Analysis ◽  
Person Identity ◽  
Identity Representation ◽  
The Brain

AbstractFace-selective and voice-selective brain regions have been shown to represent face-identity and voice-identity, respectively. Here we investigated whether there are modality-general person-identity representations in the brain that can be driven by either a face or a voice, and that invariantly represent naturalistically varying face and voice tokens of the same identity. According to two distinct models, such representations could exist either in multimodal brain regions (Campanella and Belin, 2007) or in face-selective brain regions via direct coupling between face- and voice-selective regions (von Kriegstein et al., 2005). To test the predictions of these two models, we used fMRI to measure brain activity patterns elicited by the faces and voices of familiar people in multimodal, face-selective and voice-selective brain regions. We used representational similarity analysis (RSA) to compare the representational geometries of face- and voice-elicited person-identities, and to investigate the degree to which pattern discriminants for pairs of identities generalise from one modality to the other. We found no matching geometries for faces and voices in any brain regions. However, we showed crossmodal generalisation of the pattern discriminants in the multimodal right posterior superior temporal sulcus (rpSTS), suggesting a modality-general person-identity representation in this region. Importantly, the rpSTS showed invariant representations of face- and voice-identities, in that discriminants were trained and tested on independent face videos (different viewpoint, lighting, background) and voice recordings (different vocalizations). Our findings support the Multimodal Processing Model, which proposes that face and voice information is integrated in multimodal brain regions.Significance statementIt is possible to identify a familiar person either by looking at their face or by listening to their voice. Using fMRI and representational similarity analysis (RSA) we show that the right posterior superior sulcus (rpSTS), a multimodal brain region that responds to both faces and voices, contains representations that can distinguish between familiar people independently of whether we are looking at their face or listening to their voice. Crucially, these representations generalised across different particular face videos and voice recordings. Our findings suggest that identity information from visual and auditory processing systems is combined and integrated in the multimodal rpSTS region.

scholarly journals Inferring brain-computational mechanisms with models of activity measurements

2016 ◽  
Vol 371 (1705) ◽  
pp. 20160278 ◽  
Author(s):  
Nikolaus Kriegeskorte ◽  
Jörn Diedrichsen
Keyword(s):  
Computational Models ◽  
Brain Activity ◽  
Visual Object ◽  
Visual Object Recognition ◽  
Summary Statistics ◽  
Activity Data ◽  
Individual Measurement ◽  
Activity Measurements ◽  
The Brain ◽  
The Way

High-resolution functional imaging is providing increasingly rich measurements of brain activity in animals and humans. A major challenge is to leverage such data to gain insight into the brain's computational mechanisms. The first step is to define candidate brain-computational models (BCMs) that can perform the behavioural task in question. We would then like to infer which of the candidate BCMs best accounts for measured brain-activity data. Here we describe a method that complements each BCM by a measurement model (MM), which simulates the way the brain-activity measurements reflect neuronal activity (e.g. local averaging in functional magnetic resonance imaging (fMRI) voxels or sparse sampling in array recordings). The resulting generative model (BCM-MM) produces simulated measurements. To avoid having to fit the MM to predict each individual measurement channel of the brain-activity data, we compare the measured and predicted data at the level of summary statistics. We describe a novel particular implementation of this approach, called probabilistic representational similarity analysis (pRSA) with MMs, which uses representational dissimilarity matrices (RDMs) as the summary statistics. We validate this method by simulations of fMRI measurements (locally averaging voxels) based on a deep convolutional neural network for visual object recognition. Results indicate that the way the measurements sample the activity patterns strongly affects the apparent representational dissimilarities. However, modelling of the measurement process can account for these effects, and different BCMs remain distinguishable even under substantial noise. The pRSA method enables us to perform Bayesian inference on the set of BCMs and to recognize the data-generating model in each case. This article is part of the themed issue ‘Interpreting BOLD: a dialogue between cognitive and cellular neuroscience’.

scholarly journals Harmonic Brain Modes: A Unifying Framework for Linking Space and Time in Brain Dynamics

2017 ◽  
Vol 24 (3) ◽  
pp. 277-293 ◽  
Author(s):  
Selen Atasoy ◽  
Gustavo Deco ◽  
Morten L. Kringelbach ◽  
Joel Pearson
Keyword(s):  
Neural Activity ◽  
Brain Activity ◽  
Activity Patterns ◽  
Structural Connectivity ◽  
Building Blocks ◽  
Mental States ◽  
Brain Dynamics ◽  
Space And Time ◽  
Wide Range ◽  
The Brain

A fundamental characteristic of spontaneous brain activity is coherent oscillations covering a wide range of frequencies. Interestingly, these temporal oscillations are highly correlated among spatially distributed cortical areas forming structured correlation patterns known as the resting state networks, although the brain is never truly at “rest.” Here, we introduce the concept of harmonic brain modes—fundamental building blocks of complex spatiotemporal patterns of neural activity. We define these elementary harmonic brain modes as harmonic modes of structural connectivity; that is, connectome harmonics, yielding fully synchronous neural activity patterns with different frequency oscillations emerging on and constrained by the particular structure of the brain. Hence, this particular definition implicitly links the hitherto poorly understood dimensions of space and time in brain dynamics and its underlying anatomy. Further we show how harmonic brain modes can explain the relationship between neurophysiological, temporal, and network-level changes in the brain across different mental states ( wakefulness, sleep, anesthesia, psychedelic). Notably, when decoded as activation of connectome harmonics, spatial and temporal characteristics of neural activity naturally emerge from the interplay between excitation and inhibition and this critical relation fits the spatial, temporal, and neurophysiological changes associated with different mental states. Thus, the introduced framework of harmonic brain modes not only establishes a relation between the spatial structure of correlation patterns and temporal oscillations (linking space and time in brain dynamics), but also enables a new dimension of tools for understanding fundamental principles underlying brain dynamics in different states of consciousness.

scholarly journals Neural evidence for the prediction of animacy features during language comprehension: Evidence from MEG and EEG Representational Similarity Analysis

2019 ◽  
Author(s):  
Lin Wang ◽  
Edward Wlotko ◽  
Edward Alexander ◽  
Lotte Schoot ◽  
Minjae Kim ◽  
...  
Keyword(s):  
Spatial Patterns ◽  
Language Comprehension ◽  
Neural Activity ◽  
Semantic Processing ◽  
Sentence Comprehension ◽  
Brain Activity ◽  
Coarse Grained ◽  
Similarity Analysis ◽  
Semantic Features ◽  
Representational Similarity Analysis

AbstractIt has been proposed that people can generate probabilistic predictions at multiple levels of representation during language comprehension. We used Magnetoencephalography (MEG) and Electroencephalography (EEG), in combination with Representational Similarity Analysis (RSA), to seek neural evidence for the prediction of animacy features. In two studies, MEG and EEG activity was measured as human participants (both sexes) read three-sentence scenarios. Verbs in the final sentences constrained for either animate or inanimate semantic features of upcoming nouns, and the broader discourse context constrained for either a specific noun or for multiple nouns belonging to the same animacy category. We quantified the similarity between spatial patterns of brain activity following the verbs until just before the presentation of the nouns. The MEG and EEG datasets revealed converging evidence that the similarity between spatial patterns of neural activity following animate constraining verbs was greater than following inanimate constraining verbs. This effect could not be explained by lexical-semantic processing of the verbs themselves. We therefore suggest that it reflected the inherent difference in the semantic similarity structure of the predicted animate and inanimate nouns. Moreover, the effect was present regardless of whether a specific word could be predicted, providing strong evidence for the prediction of coarse-grained semantic features that goes beyond the prediction of individual words.Significance statementLanguage inputs unfold very quickly during real-time communication. By predicting ahead we can give our brains a “head-start”, so that language comprehension is faster and more efficient. While most contexts do not constrain strongly for a specific word, they do allow us to predict some upcoming information. For example, following the context, “they cautioned the…”, we can predict that the next word will be animate rather than inanimate (we can caution a person, but not an object). Here we used EEG and MEG techniques to show that the brain is able to use these contextual constraints to predict the animacy of upcoming words during sentence comprehension, and that these predictions are associated with specific spatial patterns of neural activity.

scholarly journals Inter-subject representational similarity analysis reveals individual variations in affective experience when watching erotic movies

2019 ◽  
Author(s):  
Pin-Hao A. Chen ◽  
Eshin Jolly ◽  
Jin Hyun Cheong ◽  
Luke J. Chang
Keyword(s):  
Executive Control ◽  
The Self ◽  
Self Control ◽  
Similarity Analysis ◽  
Affective Experience ◽  
Representational Similarity Analysis ◽  
Neural Processes ◽  
Individual Variations ◽  
Affective Experiences ◽  
The Brain

AbstractWe spend much of our life pursuing or avoiding affective experiences. However, surprisingly little is known about how these experiences are represented in the brain and if they are shared across individuals. Here, we explore variations in the construction of an affective experience during a naturalistic viewing paradigm based on subjective preferences in sociosexual desire and self-control using intersubject representational similarity analysis (IS-RSA). We found that when watching erotic movies, intersubject variations in sociosexual desire preferences of 26 heterosexual males were associated with similarly structured fluctuations in the cortico-striatal reward, default mode, and mentalizing networks. In contrast, variations in the self-control preferences were associated with shared dynamics in the fronto-parietal executive control and cingulo-insula salience networks. Importantly, these results were specific to the affective experience, as we did not observe any relationship with variation in preferences when individuals watched neutral movies. Moreover, these results appear to require multivariate representations of preferences as we did not observe any significant results using single summary scores. Our findings demonstrate that multidimensional variations in individual preferences can be used to uncover unique dimensions of an affective experience, and that IS-RSA can provide new insights into the neural processes underlying psychological experiences elicited through naturalistic experimental designs.

scholarly journals Sensory-modality independent activation of the brain network for language

2019 ◽  
Author(s):  
Sophie Arana ◽  
André Marquand ◽  
Annika Hultén ◽  
Peter Hagoort ◽  
Jan-Mathijs Schoffelen
Keyword(s):  
Language Processing ◽  
Sentence Processing ◽  
Human Subjects ◽  
Brain Activity ◽  
Activity Patterns ◽  
Sensory Modality ◽  
Brain Network ◽  
Large Network ◽  
Individual Subject ◽  
The Brain

AbstractThe meaning of a sentence can be understood, whether presented in written or spoken form. Therefore it is highly probable that brain processes supporting language comprehension are at least partly independent of sensory modality. To identify where and when in the brain language processing is independent of sensory modality, we directly compared neuromagnetic brain signals of 200 human subjects (102 males) either reading or listening to sentences. We used multiset canonical correlation analysis to align individual subject data in a way that boosts those aspects of the signal that are common to all, allowing us to capture word-by-word signal variations, consistent across subjects and at a fine temporal scale. Quantifying this consistency in activation across both reading and listening tasks revealed a mostly left hemispheric cortical network. Areas showing consistent activity patterns include not only areas previously implicated in higher-level language processing, such as left prefrontal, superior & middle temporal areas and anterior temporal lobe, but also parts of the control-network as well as subcentral and more posterior temporal-parietal areas. Activity in this supramodal sentence processing network starts in temporal areas and rapidly spreads to the other regions involved. The findings do not only indicate the involvement of a large network of brain areas in supramodal language processing, but also indicate that the linguistic information contained in the unfolding sentences modulates brain activity in a word-specific manner across subjects.

scholarly journals Other people’s gaze encoded as implied motion in the human brain

2020 ◽  
Vol 117 (23) ◽  
pp. 13162-13167 ◽  
Author(s):  
Arvid Guterstam ◽  
Andrew I. Wilterson ◽  
Davis Wachtell ◽  
Michael S. A. Graziano
Keyword(s):  
Social Cognition ◽  
Theory Of Mind ◽  
Visual Motion ◽  
Brain Activity ◽  
Activity Patterns ◽  
Human Motion ◽  
Fundamental Aspect ◽  
Implied Motion ◽  
Motion System ◽  
The Brain

Keeping track of other people’s gaze is an essential task in social cognition and key for successfully reading other people’s intentions and beliefs (theory of mind). Recent behavioral evidence suggests that we construct an implicit model of other people’s gaze, which may incorporate physically incoherent attributes such as a construct of force-carrying beams that emanate from the eyes. Here, we used functional magnetic resonance imaging and multivoxel pattern analysis to test the prediction that the brain encodes gaze as implied motion streaming from an agent toward a gazed-upon object. We found that a classifier, trained to discriminate the direction of visual motion, significantly decoded the gaze direction in static images depicting a sighted face, but not a blindfolded one, from brain activity patterns in the human motion-sensitive middle temporal complex (MT+) and temporo-parietal junction (TPJ). Our results demonstrate a link between the visual motion system and social brain mechanisms, in which the TPJ, a key node in theory of mind, works in concert with MT+ to encode gaze as implied motion. This model may be a fundamental aspect of social cognition that allows us to efficiently connect agents with the objects of their attention. It is as if the brain draws a quick visual sketch with moving arrows to help keep track of who is attending to what. This implicit, fluid-flow model of other people’s gaze may help explain culturally universal myths about the mind as an energy-like, flowing essence.

scholarly journals Mirror neuron: a beautiful unnecessary concept

2021 ◽  
Vol 11 (6) ◽  
pp. 172-174
Author(s):  
Jahan N Schad
Keyword(s):  
Mirror Neurons ◽  
Brain Activity ◽  
Activity Patterns ◽  
Mirror Neuron ◽  
Prima Facie ◽  
Living Environment ◽  
Intentional Stance ◽  
Neural Structure ◽  
Tactile Input ◽  
The Brain

Mirror neurons theory, which had been put forward in the eighties based on the results of cognitive research experiments on the macaque monkeys, has prima facie been further validated by the extensive cognitive neurosciences investigations of primates and humans, over the past three decades. The concept was initially prompted by the fact that the brain activity patterns of the subjects were nearly similar, whether the activity was performed or observed by them. And presently, learning of various natures and empathy, and perhaps some aspects of survival, are ascribed to the operations of this class of neurons. Obviously the added complexity on the already complex field of neurosciences cannot be underestimated; and of course there are opponents of the theory, and some profound questions have been raised. Present work, though also in opposition, is based on completely different ground: the fact that the ingenious and grand efforts of the proponents of the theory can be explicated in the realm of the established neural structure of the brain and its computational operations. This possibility is based on the recent discovery of the tactile nature of the vision sensation. Ironically all the results, which form the basis of the mirror neuron concept, also serve to provide the conceptual proof of the new vision theory, which preempts any need for the introduction of the new class of neurons. The vision theory, partially validated through the efforts of the development of the tactile vision substitution systems (TVSS) and ironically also by some to the point mirror neuron experimental works, are sufficient to explain the processes behind empathy, learning and perhaps other mental phenomena; and as such, the need for presumption of additional class of neurons is dispelled. The mental phenomena, which rendered the claim of the mirror neurons, are simply the consequence of subjects beings variably touched by the state of the living environment, through the coherent tactile operation of all senses (four already known as having tactile nature); eyes having the most prominent role: It is the brain’s response (the computations outputs) as motor cortex activity,-- subsequent to the discernment of the streaming massive tactile input data, to appropriately coordinate the observer’s perceived (tactile) engagement, conditioned by the her mental intentional stance sourced in the brain’s protocols (acquired neural patterns)--which is misinterpreted as the evidence for the conceptualization of the mirror neuron.

scholarly journals Mapping neural activity patterns to contextualized fearful facial expressions onto callous-unemotional (CU) traits: intersubject representational similarity analysis reveals less variation among high-CU adolescents

2020 ◽  
Vol 3 ◽  
Author(s):  
Shawn A. Rhoads ◽  
Elise M. Cardinale ◽  
Katherine O’Connell ◽  
Amy L. Palmer ◽  
John W. VanMeter ◽  
...  
Keyword(s):  
Facial Expressions ◽  
Visual Processing ◽  
Activity Patterns ◽  
Neural Response ◽  
Similarity Analysis ◽  
Response Patterns ◽  
Representational Similarity Analysis ◽  
Cu Traits ◽  
Fearful Faces ◽  
Callous Unemotional

Abstract Callous-unemotional (CU) traits are early-emerging personality features characterized by deficits in empathy, concern for others, and remorse following social transgressions. One of the interpersonal deficits most consistently associated with CU traits is impaired behavioral and neurophysiological responsiveness to fearful facial expressions. However, the facial expression paradigms traditionally employed in neuroimaging are often ambiguous with respect to the nature of threat (i.e., is the perceiver the threat, or is something else in the environment?). In the present study, 30 adolescents with varying CU traits viewed fearful facial expressions cued to three different contexts (“afraid for you,” “afraid of you,” “afraid for self”) while undergoing functional magnetic resonance imaging (fMRI). Univariate analyses found that mean right amygdala activity during the “afraid for self” context was negatively associated with CU traits. With the goal of disentangling idiosyncratic stimulus-driven neural responses, we employed intersubject representational similarity analysis to link intersubject similarities in multivoxel neural response patterns to contextualized fearful expressions with differential intersubject models of CU traits. Among low-CU adolescents, neural response patterns while viewing fearful faces were most consistently similar early in the visual processing stream and among regions implicated in affective responding, but were more idiosyncratic as emotional face information moved up the cortical processing hierarchy. By contrast, high-CU adolescents’ neural response patterns consistently aligned along the entire cortical hierarchy (but diverged among low-CU youths). Observed patterns varied across contexts, suggesting that interpretations of fearful expressions depend to an extent on neural response patterns and are further shaped by levels of CU traits.

scholarly journals Representational structure or task structure? Bias in neural representational similarity analysis and a Bayesian method for reducing bias

2018 ◽  
Author(s):  
Ming Bo Cai ◽  
Nicolas W. Schuck ◽  
Jonathan W. Pillow ◽  
Yael Niv
Keyword(s):  
Neural Activity ◽  
Signal To Noise Ratio ◽  
Activity Patterns ◽  
Covariance Structure ◽  
Neural Population ◽  
Similarity Analysis ◽  
Imaging Data ◽  
Signal To Noise ◽  
Representational Similarity Analysis ◽  
Noise Ratio

AbstractThe activity of neural populations in the brains of humans and animals can exhibit vastly different spatial patterns when faced with different tasks or environmental stimuli. The degree of similarity between these neural activity patterns in response to different events is used to characterize the representational structure of cognitive states in a neural population. The dominant methods of investigating this similarity structure first estimate neural activity patterns from noisy neural imaging data using linear regression, and then examine the similarity between the estimated patterns. Here, we show that this approach introduces spurious bias structure in the resulting similarity matrix, in particular when applied to fMRI data. This problem is especially severe when the signal-to-noise ratio is low and in cases where experimental conditions cannot be fully randomized in a task. We propose Bayesian Representational Similarity Analysis (BRSA), an alternative method for computing representational similarity, in which we treat the covariance structure of neural activity patterns as a hyper-parameter in a generative model of the neural data. By marginalizing over the unknown activity patterns, we can directly estimate this covariance structure from imaging data. This method offers significant reductions in bias and allows estimation of neural representational similarity with previously unattained levels of precision at low signal-to-noise ratio. The probabilistic framework allows for jointly analyzing data from a group of participants. The method can also simultaneously estimate a signal-to-noise ratio map that shows where the learned representational structure is supported more strongly. Both this map and the learned covariance matrix can be used as a structured prior for maximum a posteriori estimation of neural activity patterns, which can be further used for fMRI decoding. We make our tool freely available in Brain Imaging Analysis Kit (BrainIAK).Author summaryWe show the severity of the bias introduced when performing representational similarity analysis (RSA) based on neural activity pattern estimated within imaging runs. Our Bayesian RSA method significantly reduces the bias and can learn a shared representational structure across multiple participants. We also demonstrate its extension as a new multi-class decoding tool.

Sign in / Sign up

Export Citation Format

Share Document