scholarly journals Visually Grounded and Textual Semantic Models Differentially Decode Brain Activity Associated with Concrete and Abstract Nouns

2017 ◽  
Vol 5 ◽  
pp. 17-30 ◽  
Author(s):  
Andrew J. Anderson ◽  
Douwe Kiela ◽  
Stephen Clark ◽  
Massimo Poesio
Keyword(s):  
Coding Theory ◽  
Computational Models ◽  
Brain Activity ◽  
Activity Patterns ◽  
Abstract Concepts ◽  
Deep Convolutional Neural Networks ◽  
Abstract Nouns ◽  
Semantic Models ◽  
Concrete Nouns ◽  
The Brain

Important advances have recently been made using computational semantic models to decode brain activity patterns associated with concepts; however, this work has almost exclusively focused on concrete nouns. How well these models extend to decoding abstract nouns is largely unknown. We address this question by applying state-of-the-art computational models to decode functional Magnetic Resonance Imaging (fMRI) activity patterns, elicited by participants reading and imagining a diverse set of both concrete and abstract nouns. One of the models we use is linguistic, exploiting the recent word2vec skipgram approach trained on Wikipedia. The second is visually grounded, using deep convolutional neural networks trained on Google Images. Dual coding theory considers concrete concepts to be encoded in the brain both linguistically and visually, and abstract concepts only linguistically. Splitting the fMRI data according to human concreteness ratings, we indeed observe that both models significantly decode the most concrete nouns; however, accuracy is significantly greater using the text-based models for the most abstract nouns. More generally this confirms that current computational models are sufficiently advanced to assist in investigating the representational structure of abstract concepts in the brain.

scholarly journals Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models

2020 ◽  
Vol 8 ◽  
pp. 231-246
Author(s):  
Vesna G. Djokic ◽  
Jean Maillard ◽  
Luana Bulat ◽  
Ekaterina Shutova
Keyword(s):  
Language Processing ◽  
Sentence Comprehension ◽  
Brain Activity ◽  
Activity Patterns ◽  
Word Embeddings ◽  
Semantic Models ◽  
Distributional Semantic Models ◽  
Metaphor Processing ◽  
Meaning Representation ◽  
The Brain

Recent years have seen a growing interest within the natural language processing (NLP) community in evaluating the ability of semantic models to capture human meaning representation in the brain. Existing research has mainly focused on applying semantic models to decode brain activity patterns associated with the meaning of individual words, and, more recently, this approach has been extended to sentences and larger text fragments. Our work is the first to investigate metaphor processing in the brain in this context. We evaluate a range of semantic models (word embeddings, compositional, and visual models) in their ability to decode brain activity associated with reading of both literal and metaphoric sentences. Our results suggest that compositional models and word embeddings are able to capture differences in the processing of literal and metaphoric sentences, providing support for the idea that the literal meaning is not fully accessible during familiar metaphor comprehension.

scholarly journals Communicating abstract meaning: concepts revealed in words and gestures

2018 ◽  
Vol 373 (1752) ◽  
pp. 20170138 ◽  
Author(s):  
Lenka Zdrazilova ◽  
David M. Sidhu ◽  
Penny M. Pexman
Keyword(s):  
Concrete Word ◽  
Abstract Word ◽  
Abstract Concepts ◽  
Multiple Representation ◽  
Word Meanings ◽  
Representation Systems ◽  
Abstract Nouns ◽  
Concrete Nouns ◽  
Abstract Words ◽  
The Brain

Abstract words refer to concepts that cannot be directly experienced through our senses (e.g. truth , morality ). How we ground the meanings of abstract words is one of the deepest problems in cognitive science today. We investigated this question in an experiment in which 62 participants were asked to communicate the meanings of words (20 abstract nouns, e.g. impulse ; 10 concrete nouns, e.g. insect ) to a partner without using the words themselves (the taboo task). We analysed the speech and associated gestures that participants used to communicate the meaning of each word in the taboo task. Analysis of verbal and gestural data yielded a number of insights. When communicating about the meanings of abstract words, participants' speech referenced more people and introspections. In contrast, the meanings of concrete words were communicated by referencing more objects and entities. Gesture results showed that when participants spoke about abstract word meanings their speech was accompanied by more metaphorical and beat gestures, and speech about concrete word meanings was accompanied by more iconic gestures. Taken together, the results suggest that abstract meanings are best captured by a model that allows dynamic access to multiple representation systems. This article is part of the theme issue ‘Varieties of abstract concepts: development, use and representation in the brain’.

scholarly journals Modelling brain representations of abstract concepts

2021 ◽  
Author(s):  
Daniel Kaiser ◽  
Arthur Jacobs ◽  
Radoslaw Martin Cichy
Keyword(s):  
Computational Models ◽  
Activity Patterns ◽  
Human Cognition ◽  
Distributional Semantics ◽  
Abstract Concepts ◽  
Conceptual Representations ◽  
Training Materials ◽  
Abstract Nouns ◽  
Inferior Parietal Cortex ◽  
Background Context

Abstract conceptual representations are critical for human cognition. Despite their importance, key properties of these representations remain poorly understood. Here, we used computational models of distributional semantics to predict multivariate fMRI activity patterns during the activation and contextualization of abstract concepts. We devised a task in which participants had to embed abstract nouns into a story that they developed around a given background context. We found that representations in inferior parietal cortex were predicted by concept similarities emerging in models of distributional semantics. By constructing different model families, we reveal the models' learning trajectories and delineate how abstract and concrete training materials contribute to the formation of brain-like representations. These results inform theories about the format and emergence of abstract conceptual representations in the human brain.

scholarly journals Inferring brain-computational mechanisms with models of activity measurements

2016 ◽  
Vol 371 (1705) ◽  
pp. 20160278 ◽  
Author(s):  
Nikolaus Kriegeskorte ◽  
Jörn Diedrichsen
Keyword(s):  
Computational Models ◽  
Brain Activity ◽  
Visual Object ◽  
Visual Object Recognition ◽  
Summary Statistics ◽  
Activity Data ◽  
Individual Measurement ◽  
Activity Measurements ◽  
The Brain ◽  
The Way

High-resolution functional imaging is providing increasingly rich measurements of brain activity in animals and humans. A major challenge is to leverage such data to gain insight into the brain's computational mechanisms. The first step is to define candidate brain-computational models (BCMs) that can perform the behavioural task in question. We would then like to infer which of the candidate BCMs best accounts for measured brain-activity data. Here we describe a method that complements each BCM by a measurement model (MM), which simulates the way the brain-activity measurements reflect neuronal activity (e.g. local averaging in functional magnetic resonance imaging (fMRI) voxels or sparse sampling in array recordings). The resulting generative model (BCM-MM) produces simulated measurements. To avoid having to fit the MM to predict each individual measurement channel of the brain-activity data, we compare the measured and predicted data at the level of summary statistics. We describe a novel particular implementation of this approach, called probabilistic representational similarity analysis (pRSA) with MMs, which uses representational dissimilarity matrices (RDMs) as the summary statistics. We validate this method by simulations of fMRI measurements (locally averaging voxels) based on a deep convolutional neural network for visual object recognition. Results indicate that the way the measurements sample the activity patterns strongly affects the apparent representational dissimilarities. However, modelling of the measurement process can account for these effects, and different BCMs remain distinguishable even under substantial noise. The pRSA method enables us to perform Bayesian inference on the set of BCMs and to recognize the data-generating model in each case. This article is part of the themed issue ‘Interpreting BOLD: a dialogue between cognitive and cellular neuroscience’.

scholarly journals Harmonic Brain Modes: A Unifying Framework for Linking Space and Time in Brain Dynamics

2017 ◽  
Vol 24 (3) ◽  
pp. 277-293 ◽  
Author(s):  
Selen Atasoy ◽  
Gustavo Deco ◽  
Morten L. Kringelbach ◽  
Joel Pearson
Keyword(s):  
Neural Activity ◽  
Brain Activity ◽  
Activity Patterns ◽  
Structural Connectivity ◽  
Building Blocks ◽  
Mental States ◽  
Brain Dynamics ◽  
Space And Time ◽  
Wide Range ◽  
The Brain

A fundamental characteristic of spontaneous brain activity is coherent oscillations covering a wide range of frequencies. Interestingly, these temporal oscillations are highly correlated among spatially distributed cortical areas forming structured correlation patterns known as the resting state networks, although the brain is never truly at “rest.” Here, we introduce the concept of harmonic brain modes—fundamental building blocks of complex spatiotemporal patterns of neural activity. We define these elementary harmonic brain modes as harmonic modes of structural connectivity; that is, connectome harmonics, yielding fully synchronous neural activity patterns with different frequency oscillations emerging on and constrained by the particular structure of the brain. Hence, this particular definition implicitly links the hitherto poorly understood dimensions of space and time in brain dynamics and its underlying anatomy. Further we show how harmonic brain modes can explain the relationship between neurophysiological, temporal, and network-level changes in the brain across different mental states ( wakefulness, sleep, anesthesia, psychedelic). Notably, when decoded as activation of connectome harmonics, spatial and temporal characteristics of neural activity naturally emerge from the interplay between excitation and inhibition and this critical relation fits the spatial, temporal, and neurophysiological changes associated with different mental states. Thus, the introduced framework of harmonic brain modes not only establishes a relation between the spatial structure of correlation patterns and temporal oscillations (linking space and time in brain dynamics), but also enables a new dimension of tools for understanding fundamental principles underlying brain dynamics in different states of consciousness.

scholarly journals Transient brain networks underlying interpersonal strategies during synchronized action

2020 ◽  
Author(s):  
Ole Adrian Heggli ◽  
Ivana Konvalinka ◽  
Joana Cabral ◽  
Elvira Brattico ◽  
Morten L Kringelbach ◽  
...  
Keyword(s):  
Computational Models ◽  
Brain Activity ◽  
Brain Regions ◽  
Brain Network ◽  
Human Interaction ◽  
Action Perception ◽  
Network Information ◽  
Mutual Adaptation ◽  
Underlying Mechanisms ◽  
The Brain

Abstract Interpersonal coordination is a core part of human interaction, and its underlying mechanisms have been extensively studied using social paradigms such as joint finger-tapping. Here, individual and dyadic differences have been found to yield a range of dyadic synchronization strategies, such as mutual adaptation, leading–leading, and leading–following behaviour, but the brain mechanisms that underlie these strategies remain poorly understood. To identify individual brain mechanisms underlying emergence of these minimal social interaction strategies, we contrasted EEG-recorded brain activity in two groups of musicians exhibiting the mutual adaptation and leading–leading strategies. We found that the individuals coordinating via mutual adaptation exhibited a more frequent occurrence of phase-locked activity within a transient action–perception-related brain network in the alpha range, as compared to the leading–leading group. Furthermore, we identified parietal and temporal brain regions that changed significantly in the directionality of their within-network information flow. Our results suggest that the stronger weight on extrinsic coupling observed in computational models of mutual adaptation as compared to leading–leading might be facilitated by a higher degree of action–perception network coupling in the brain.

Creativity or mental illness: Possible errors of relational priming in neural networks of the brain

2008 ◽  
Vol 31 (4) ◽  
pp. 398-399
Author(s):  
James E. Swain ◽  
John D. Swain
Keyword(s):  
Mental Illness ◽  
Cognitive Skills ◽  
Computational Models ◽  
Brain Activity ◽  
Childhood Onset ◽  
Hyperactivity Disorder ◽  
Depression Disorders ◽  
Complex Cognitive Skills ◽  
The Brain ◽  
Relational Priming

AbstractIf connectionist computational models explain the acquisition of complex cognitive skills, errors in such models would also help explain unusual brain activity such as in creativity – as well as in mental illness, including childhood onset problems with social behaviors in autism, the inability to maintain focus in attention deficit and hyperactivity disorder (ADHD), and the lack of motivation of depression disorders.

scholarly journals Searching through functional space reveals distributed visual, auditory, and semantic coding in the human brain

2020 ◽  
Author(s):  
Sreejan Kumar ◽  
Cameron T. Ellis ◽  
Thomas O’Connell ◽  
Marvin M Chun ◽  
Nicholas B. Turk-Browne
Keyword(s):  
Human Brain ◽  
Language Processing ◽  
Brain Function ◽  
Computational Models ◽  
Brain Activity ◽  
Functional Space ◽  
Semantic Features ◽  
Brain Functions ◽  
Auditory Features ◽  
The Brain

AbstractThe extent to which brain functions are localized or distributed is a foundational question in neuroscience. In the human brain, common fMRI methods such as cluster correction, atlas parcellation, and anatomical searchlight are biased by design toward finding localized representations. Here we introduce the functional searchlight approach as an alternative to anatomical searchlight analysis, the most commonly used exploratory multivariate fMRI technique. Functional searchlight removes any anatomical bias by grouping voxels based only on functional similarity and ignoring anatomical proximity. We report evidence that visual and auditory features from deep neural networks and semantic features from a natural language processing model are more widely distributed across the brain than previously acknowledged. This approach provides a new way to evaluate and constrain computational models with brain activity and pushes our understanding of human brain function further along the spectrum from strict modularity toward distributed representation.

scholarly journals Feature-reweighted RSA: A method for improving the fit between computational models, brains, and behavior

2021 ◽  
Author(s):  
Philipp Kaniuth ◽  
Martin N. Hebart
Keyword(s):  
Model Selection ◽  
Computational Models ◽  
Model Comparison ◽  
Brain Activity ◽  
Activity Patterns ◽  
Behavioral Data ◽  
Novel Approach ◽  
Improve Model ◽  
And Behavior ◽  
Model Features

AbstractRepresentational Similarity Analysis (RSA) has emerged as a popular method for relating representational spaces from human brain activity, behavioral data, and computational models. RSA is based on the comparison of representational dissimilarity matrices (RDM), which characterize the pairwise dissimilarities of all conditions across all features (e.g. fMRI voxels or units of a model). However, classical RSA treats each feature as equally important. This ‘equal weights’ assumption contrasts with the flexibility of multivariate decoding, which reweights individual features for predicting a target variable. As a consequence, classical RSA may lead researchers to underestimate the correspondence between a model and a brain region and, for model comparison, it may lead to selecting the inferior model. While previous work has suggested that reweighting can improve model selection in RSA, it has remained unclear to what extent these results generalize across datasets and data modalities. To fill this gap, the aim of this work is twofold: First, utilizing a range of publicly available datasets and three popular deep neural networks (DNNs), we seek to broadly test feature-reweighted RSA (FR-RSA) applied to computational models and reveal the extent to which reweighting model features improves RDM correspondence and affects model selection. Second, we propose voxel-reweighted RSA, a novel use case of FR-RSA that reweights fMRI voxels, mirroring the rationale of multivariate decoding of optimally combining voxel activity patterns. We find that reweighting individual model units (1) markedly improves the fit between model RDMs and target RDMs derived from several fMRI and behavioral datasets and (2) affects model selection, highlighting the importance of considering FR-RSA. For voxel-reweighted RSA, improvements in RDM correspondence were even more pronounced, demonstrating the utility of this novel approach. We additionally demonstrate that classical noise ceilings can be exceeded when FR-RSA is applied and propose an updated approach for their computation. Taken together, our results broadly validate the use of FR-RSA for improving the fit between computational models, brain and behavioral data, possibly allowing us to better adjudicate between competing computational models. Further, our results suggest that FR-RSA applied to brain measurement channels could become an important new method to assess the match between representational spaces.

scholarly journals Sensory-modality independent activation of the brain network for language

2019 ◽  
Author(s):  
Sophie Arana ◽  
André Marquand ◽  
Annika Hultén ◽  
Peter Hagoort ◽  
Jan-Mathijs Schoffelen
Keyword(s):  
Language Processing ◽  
Sentence Processing ◽  
Human Subjects ◽  
Brain Activity ◽  
Activity Patterns ◽  
Sensory Modality ◽  
Brain Network ◽  
Large Network ◽  
Individual Subject ◽  
The Brain

AbstractThe meaning of a sentence can be understood, whether presented in written or spoken form. Therefore it is highly probable that brain processes supporting language comprehension are at least partly independent of sensory modality. To identify where and when in the brain language processing is independent of sensory modality, we directly compared neuromagnetic brain signals of 200 human subjects (102 males) either reading or listening to sentences. We used multiset canonical correlation analysis to align individual subject data in a way that boosts those aspects of the signal that are common to all, allowing us to capture word-by-word signal variations, consistent across subjects and at a fine temporal scale. Quantifying this consistency in activation across both reading and listening tasks revealed a mostly left hemispheric cortical network. Areas showing consistent activity patterns include not only areas previously implicated in higher-level language processing, such as left prefrontal, superior & middle temporal areas and anterior temporal lobe, but also parts of the control-network as well as subcentral and more posterior temporal-parietal areas. Activity in this supramodal sentence processing network starts in temporal areas and rapidly spreads to the other regions involved. The findings do not only indicate the involvement of a large network of brain areas in supramodal language processing, but also indicate that the linguistic information contained in the unfolding sentences modulates brain activity in a word-specific manner across subjects.

Sign in / Sign up

Export Citation Format

Share Document