Re-evaluating visual and auditory dominance through modality switching costs and congruency analyses

Rajwant Sandhu; Benjamin J. Dyson

doi:10.1016/j.actpsy.2012.04.003

The modality switching costs of Chinese–English bilinguals in the processing of L1 and L2

Quarterly Journal of Experimental Psychology ◽

10.1177/1747021819878089 ◽

2019 ◽

Vol 73 (3) ◽

pp. 396-412

Author(s):

Tianyang Zhao ◽

Yanli Huang ◽

Donggui Chen ◽

Lu Jiao ◽

Fernando Marmolejo-Ramos ◽

...

Keyword(s):

Language Comprehension ◽

Language Production ◽

Switching Costs ◽

Naming Task ◽

Visual Modality ◽

Decision Task ◽

Auditory Modality ◽

Sequential Information ◽

Modality Switching ◽

L1 And L2

Modality switching cost indicates that people’s performance becomes worse when they judge sequential information that is related to different sensory modalities than judging information that is related to the same modality. In this study, we conducted three experiments on proficient and non-proficient bilingual individuals to investigate the modality switching costs in L1 and L2 processing separately. In Experiment 1, materials were L1 and L2 words that were either conceptually related to a visual modality (e.g., light) or related to an auditory modality (e.g., song). The modality switching costs were investigated in a lexical decision task in both L1 and L2. Experiment 2 further explored the modality switching costs while weakening the activation level of the perceptual modality by adding a set of fillers. Experiment 3 used a word-naming task to explore the modality switching effect in language production in L1 and L2. Results of these experiments showed that the modality switching costs appeared in both language comprehension and production in L1 and L2 conditions. The magnitude of the modality switching costs was conditionally modulated by the L2 proficiency level, such as in the L2 condition in Experiment 1 and in both L1 and L2 conditions in Experiment 3. These results suggest that sensorimotor simulation is involved in not only language comprehension but also language production. The sensorimotor simulation that is acquired in L1 can be transferred to L2.

Download Full-text

Investigating task and modality switching costs using bimodal stimuli

Seeing and Perceiving ◽

10.1163/187847612x646451 ◽

2012 ◽

Vol 25 (0) ◽

pp. 22

Author(s):

Raj Sandhu ◽

Ben Dyson

Keyword(s):

Task Switching ◽

Visual Stimulus ◽

Switching Costs ◽

Stimulus Presentation ◽

Task Type ◽

Task Repetition ◽

Specific Combination ◽

Task Change ◽

Modality Switching ◽

And Task

Investigations of concurrent task and modality switching effects have to date been studied under conditions of uni-modal stimulus presentation. As such, it is difficult to directly compare resultant task and modality switching effects, as the stimuli afford both tasks on each trial, but only one modality. The current study investigated task and modality switching using bi-modal stimulus presentation under various cue conditions: task and modality (double cue), either task or modality (single cue) or no cue. Participants responded to either the identity or the position of an audio–visual stimulus. Switching effects were defined as staying within a modality/task (repetition) or switching into a modality/task (change) from trial n − 1 to trial n, with analysis performed on trial n data. While task and modality switching costs were sub-additive across all conditions replicating previous data, modality switching effects were dependent on the modality being attended, and task switching effects were dependent on the task being performed. Specifically, visual responding and position responding revealed significant costs associated with modality and task switching, while auditory responding and identity responding revealed significant gains associated with modality and task switching. The effects interacted further, revealing that costs and gains associated with task and modality switching varying with the specific combination of modality and task type. The current study reconciles previous data by suggesting that efficiently processed modality/task information benefits from repetition while less efficiently processed information benefits from change due to less interference of preferred processing across consecutive trials.

Download Full-text

Continuous neural activations of embodied-linguistic representations in modality switching costs

10.31234/osf.io/tkcpx ◽

2021 ◽

Author(s):

Pei Q Liu ◽

Louise Connell ◽

Dermot Lynott

Keyword(s):

Switching Costs ◽

Semantic Representation ◽

Distributional Pattern ◽

N400 Effect ◽

Sensory Property ◽

Conceptual Representations ◽

Linguistic Model ◽

Linguistic Representations ◽

Perceptual Modalities ◽

Modality Switching

Modality switching costs (MSCs) are one of the classic effects that support the embodied views of conceptual representations. They refer to a delay in response time to verify a sensory property of a certain perceptual modality (e.g., visual: SUN – bright), when the previous sensory property has been of a different modality (e.g., auditory: BLENDER – loud) compared to a property of the same modality (e.g., visual: ROSE – red). Such costs indicate that conceptual representations require the recruitment of modality-specific resources. However, MSCs could also result from the distributional pattern of property words: the reason why loud -> bright takes longer than red -> bright could be because bright and loud do not co-occur in the same linguistic context as frequently as bright and red. In the present study, we examined how well MSCs were predicted by an embodied model (switch / no-switch between perceptual modalities) versus a linguistic model (switch / no-switch between linguistic distributional clusters), in behavioural (RT) and continuous event-related EEG potentials (ERP) paradigms. The behavioural data supported the linguistic model in explaining MSCs and found MSCs to be moderated by the target modality, the ERPs showed that linguistic distributional pattern played a crucial role in the neural activations of MSCs. What used to be found as a result of perceptual switching (e.g., "early N400" effect) could be better explained by the linguistic model. The embodied component was activated later than the linguistic component, accounting for activations associated with semantic representation (typically in N400 area). Later during processing, both components were active for decision making (often manifested as LPC).

Download Full-text