Auditory speech processing is affected by visual speech in the periphery

2011 ◽  
Author(s):  
Jeesun Kim ◽  
Chris Davis
Keyword(s):  
Speech Processing ◽  
Visual Speech ◽  
Auditory Speech

scholarly journals Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility

eLife ◽  
2016 ◽  
Vol 5 ◽  
Author(s):  
Hyojin Park ◽  
Christoph Kayser ◽  
Gregor Thut ◽  
Joachim Gross
Keyword(s):  
Visual Cortex ◽  
Speech Processing ◽  
Speech Intelligibility ◽  
Brain Activity ◽  
Low Frequency ◽  
Visual Speech ◽  
Visual Signals ◽  
Partial Coherence ◽  
Auditory Speech ◽  
Oscillatory Brain Activity

During continuous speech, lip movements provide visual temporal signals that facilitate speech processing. Here, using MEG we directly investigated how these visual signals interact with rhythmic brain activity in participants listening to and seeing the speaker. First, we investigated coherence between oscillatory brain activity and speaker’s lip movements and demonstrated significant entrainment in visual cortex. We then used partial coherence to remove contributions of the coherent auditory speech signal from the lip-brain coherence. Comparing this synchronization between different attention conditions revealed that attending visual speech enhances the coherence between activity in visual cortex and the speaker’s lips. Further, we identified a significant partial coherence between left motor cortex and lip movements and this partial coherence directly predicted comprehension accuracy. Our results emphasize the importance of visually entrained and attention-modulated rhythmic brain activity for the enhancement of audiovisual speech processing.

scholarly journals Reading Fluent Speech from Talking Faces: Typical Brain Networks and Individual Differences

2005 ◽  
Vol 17 (6) ◽  
pp. 939-953 ◽  
Author(s):  
Deborah A. Hall ◽  
Clayton Fussell ◽  
A. Quentin Summerfield
Keyword(s):  
Individual Differences ◽  
Speech Processing ◽  
Superior Temporal Gyrus ◽  
Brain Regions ◽  
Receptive Language ◽  
Visual Input ◽  
Visual Speech ◽  
Middle Temporal ◽  
Blank Screen ◽  
Auditory Speech

Listeners are able to extract important linguistic information by viewing the talker's face—a process known as “speechreading.” Previous studies of speechreading present small closed sets of simple words and their results indicate that visual speech processing engages a wide network of brain regions in the temporal, frontal, and parietal lobes that are likely to underlie multiple stages of the receptive language system. The present study further explored this network in a large group of subjects by presenting naturally spoken sentences which tap the richer complexities of visual speech processing. Four different baselines (blank screen, static face, nonlinguistic facial gurning, and auditory speech) enabled us to determine the hierarchy of neural processing involved in speechreading and to test the claim that visual input reliably accesses sound-based representations in the auditory cortex. In contrast to passively viewing a blank screen, the static-face condition evoked activation bilaterally across the border of the fusiform gyrus and cerebellum, and in the medial superior frontal gyrus and left precentral gyrus (p < .05, whole brain corrected). With the static face as baseline, the gurning face evoked bilateral activation in the motion-sensitive region of the occipital cortex, whereas visual speech additionally engaged the middle temporal gyrus, inferior and middle frontal gyri, and the inferior parietal lobe, particularly in the left hemisphere. These latter regions are implicated in lexical stages of spoken language processing. Although auditory speech generated extensive bilateral activation across both superior and middle temporal gyri, the group-averaged pattern of speechreading activation failed to include any auditory regions along the superior temporal gyrus, suggesting that fluent visual speech does not always involve sound-based coding of the visual input. An important finding from the individual subject analyses was that activation in the superior temporal gyrus did reach significance (p < .001, small-volume corrected) for a subset of the group. Moreover, the extent of the left-sided superior temporal gyrus activity was strongly correlated with speech-reading performance. Skilled speechreading was also associated with activations and deactivations in other brain regions, suggesting that individual differences reflect the efficiency of a circuit linking sensory, perceptual, memory, cognitive, and linguistic processes rather than the operation of a single component process.

scholarly journals Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

2020 ◽  
Author(s):  
Brian A. Metzger ◽  
John F. Magnotti ◽  
Zhengjia Wang ◽  
Elizabeth Nesbitt ◽  
Patrick J. Karas ◽  
...  
Keyword(s):  
Speech Perception ◽  
Speech Processing ◽  
Time Course ◽  
Brain Area ◽  
Superior Temporal Gyrus ◽  
Visual Speech ◽  
Audiovisual Speech ◽  
Neural Responses ◽  
Auditory Speech ◽  
Human Epilepsy

AbstractExperimentalists studying multisensory integration compare neural responses to multisensory stimuli with responses to the component modalities presented in isolation. This procedure is problematic for multisensory speech perception since audiovisual speech and auditory-only speech are easily intelligible but visual-only speech is not. To overcome this confound, we developed intracranial encephalography (iEEG) deconvolution. Individual stimuli always contained both auditory and visual speech but jittering the onset asynchrony between modalities allowed for the time course of the unisensory responses and the interaction between them to be independently estimated. We applied this procedure to electrodes implanted in human epilepsy patients (both male and female) over the posterior superior temporal gyrus (pSTG), a brain area known to be important for speech perception. iEEG deconvolution revealed sustained, positive responses to visual-only speech and larger, phasic responses to auditory-only speech. Confirming results from scalp EEG, responses to audiovisual speech were weaker than responses to auditory- only speech, demonstrating a subadditive multisensory neural computation. Leveraging the spatial resolution of iEEG, we extended these results to show that subadditivity is most pronounced in more posterior aspects of the pSTG. Across electrodes, subadditivity correlated with visual responsiveness, supporting a model in visual speech enhances the efficiency of auditory speech processing in pSTG. The ability to separate neural processes may make iEEG deconvolution useful for studying a variety of complex cognitive and perceptual tasks.Significance statementUnderstanding speech is one of the most important human abilities. Speech perception uses information from both the auditory and visual modalities. It has been difficult to study neural responses to visual speech because visual-only speech is difficult or impossible to comprehend, unlike auditory-only and audiovisual speech. We used intracranial encephalography (iEEG) deconvolution to overcome this obstacle. We found that visual speech evokes a positive response in the human posterior superior temporal gyrus, enhancing the efficiency of auditory speech processing.

scholarly journals Converging Evidence from Electrocorticography and BOLD fMRI for a Sharp Functional Boundary in Superior Temporal Gyrus Related to Multisensory Speech Processing

2018 ◽  
Author(s):  
Muge Ozker ◽  
Michael S. Beauchamp
Keyword(s):  
Speech Perception ◽  
Speech Processing ◽  
Recognition Task ◽  
Superior Temporal Gyrus ◽  
Visual Speech ◽  
Audiovisual Speech ◽  
Auditory Modality ◽  
Bold Fmri ◽  
Auditory Speech ◽  
Functional Boundary

AbstractAlthough humans can understand speech using the auditory modality alone, in noisy environments visual speech information from the talker’s mouth can rescue otherwise unintelligible auditory speech. To investigate the neural substrates of multisensory speech perception, we recorded neural activity from the human superior temporal gyrus using two very different techniques: either directly, using surface electrodes implanted in five participants with epilepsy (electrocorticography, ECOG), or indirectly, using blood oxygen level dependent functional magnetic resonance imaging (BOLD fMRI) in six healthy control fMRI participants. Both ECOG and fMRI participants viewed the same clear and noisy audiovisual speech stimuli and performed the same speech recognition task. Both techniques demonstrated a sharp functional boundary in the STG, which corresponded to an anatomical boundary defined by the posterior edge of Heschl’s gyrus. On the anterior side of the boundary, cortex responded more strongly to clear audiovisual speech than to noisy audiovisual speech, suggesting that anterior STG is primarily involved in processing unisensory auditory speech. On the posterior side of the boundary, cortex preferred noisy audiovisual speech or showed no preference and showed robust responses to auditory-only and visual-only speech, suggesting that posterior STG is specialized for processing multisensory audiovisual speech. For both ECOG and fMRI, the transition between the functionally distinct regions happened within 10 mm of anterior-to-posterior distance along the STG. We relate this boundary to the multisensory neural code underlying speech perception and propose that it represents an important functional division within the human speech perception network.

Visual speech form influences the speed of auditory speech processing

2013 ◽  
Vol 126 (3) ◽  
pp. 350-356 ◽  
Author(s):  
Tim Paris ◽  
Jeesun Kim ◽  
Chris Davis
Keyword(s):  
Speech Processing ◽  
Visual Speech ◽  
Auditory Speech

scholarly journals Somatosensory contribution to audio-visual speech processing

Cortex ◽  
2021 ◽  
Author(s):  
Takayuki Ito ◽  
Hiroki Ohashi ◽  
Vincent L. Gracco
Keyword(s):  
Speech Processing ◽  
Visual Speech
Sign in / Sign up

Export Citation Format

Share Document