Crossmodal Associations with Olfactory, Auditory, and Tactile Stimuli in Children and Adults

People associate information with different senses but the mechanism by which this happens is unclear. Such associations are thought to arise from innate structural associations in the brain, statistical associations in the environment, via shared affective content, or through language. A developmental perspective on crossmodal associations can help determine which explanations are more likely for specific associations. Certain associations with pitch (e.g., pitch–height) have been observed early in infancy, but others may only occur late into childhood (e.g., pitch–size). In contrast, tactile–chroma associations have been observed in children, but not adults. One modality that has received little attention developmentally is olfaction. In the present investigation, we explored crossmodal associations from sound, tactile stimuli, and odor to a range of stimuli by testing a broad range of participants. Across the three modalities, we found little evidence for crossmodal associations in young children. This suggests an account based on innate structures is unlikely. Instead, the number and strength of associations increased over the lifespan. This suggests that experience plays a crucial role in crossmodal associations from sound, touch, and smell to other senses.

Predicting the colour associated with odours using an electronic nose

Predicting olfactory perception with an electronic nose can aid in the design and evaluation of olfactory-based experiences. We investigate whether the human perception of odours can be predicted outside the bounds of perceived pleasantness and semantic descriptors. We tuned an electronic nose to predict an odour's colour in the CIELAB colour space using human judgements. This revealed that the crossmodal associations people have towards colours could be predicted. Our electronic nose system can predict an odour's colour with a 70 – 81% machine-human similarity rating. These findings suggest a systematic and predictable link exists between the chemical features of odours and the colour associated to them. These findings highlight the possibilities of predicting human olfactory perception using an electronic nose.

Crossmodal associations modulate multisensory spatial integration

According to the Bayesian framework of multisensory integration, audiovisual stimuli associated with a stronger prior belief that they share a common cause (i.e., causal prior) are predicted to result in a greater degree of perceptual binding and therefore greater audiovisual integration. In the present psychophysical study, we systematically manipulated the causal prior while keeping sensory evidence constant. We paired auditory and visual stimuli during an association phase to be spatiotemporally either congruent or incongruent, with the goal of driving the causal prior in opposite directions for different audiovisual pairs. Following this association phase, every pairwise combination of the auditory and visual stimuli was tested in a typical ventriloquism-effect (VE) paradigm. The size of the VE (i.e., the shift of auditory localization towards the spatially discrepant visual stimulus) indicated the degree of multisensory integration. Results showed that exposure to an audiovisual pairing as spatiotemporally congruent compared to incongruent resulted in a larger subsequent VE (Experiment 1). This effect was further confirmed in a second VE paradigm, where the congruent and the incongruent visual stimuli flanked the auditory stimulus, and a VE in the direction of the congruent visual stimulus was shown (Experiment 2). Since the unisensory reliabilities for the auditory or visual components did not change after the association phase, the observed effects are likely due to changes in multisensory binding by association learning. As suggested by Bayesian theories of multisensory processing, our findings support the existence of crossmodal causal priors that are flexibly shaped by experience in a changing world.

Assessing the Role of Emotional Mediation in Explaining Crossmodal Correspondences Involving Musical Stimuli

A wide variety of crossmodal correspondences, defined as the often surprising connections that people appear to experience between simple features, attributes, or dimensions of experience, either physically present or else merely imagined, in different sensory modalities, have been demonstrated in recent years. However, a number of crossmodal correspondences have also been documented between more complex (i.e., multi-component) stimuli, such as, for example, pieces of music and paintings. In this review, the extensive evidence supporting the emotional mediation account of the crossmodal correspondences between musical stimuli (mostly pre-recorded short classical music excerpts) and visual stimuli, including colour patches through to, on occasion, paintings, is critically evaluated. According to the emotional mediation account, it is the emotional associations that people have with stimuli that constitutes one of the fundamental bases on which crossmodal associations are established. Taken together, the literature that has been published to date supports emotional mediation as one of the key factors underlying the crossmodal correspondences involving emotionally-valenced stimuli, both simple and complex.

Crossmodal Associations between Cinema with Elements of Comic and Tragic and Texture Touch

Crossmodal associations are systematic links between a stimulus from one modality (e.g., color) and a stimulus from another modality (e.g., sound). The present study investigated crossmodal associations between cinema (a complex visual and auditory stimulus) and texture touch. Participants watched fragments of movies with comic and tragic elements. They also touched different textures (e.g., silk, fur, marble) and chose the textures that were most/least consistent with the films. We found systematic links between the films and the textures associated with them. Films with elements of tragedy were associated with granite, marble and glass, while films with elements of comedy were associated with glass pebbles, plasticine and a slime toy. Similar emotional and semantic evaluations of textures and films can partly explain these crossmodal associations. Significant correlations were found between the direct emotional evaluation of films and indirect evaluation through the cinema-haptic perception index for the following scales: dirty, disgusting, pleasant and happy.

Crossmodal Association Between Linguistic Sounds and Motion Imagery: Voicing in Obstruents Connects With Different Strengths of Motor Execution

The present study investigates whether obstruent voicing may or may not affect the imagery of different strengths of motor execution. In a modified version of the implicit association test, participants responded to discrimination tasks that include viewing static pictures of athletes in motion and hearing mono-syllabic linguistic sounds. The results suggest that voiced obstruents are compatible with the motion imagery that implies stronger motor executions, whereas voiceless obstruents are compatible with the imagery that implies weaker motor executions. These results provide experimental support for crossmodal associations between the auditory perception of linguistic sounds, namely, the voicing of obstruents, and the visually induced imagery of different levels of strength in motor actions.

The Effect of Odor and Color on Chemical Cooling

Chemesthesis, along with taste and olfaction, is a primary component of flavor that engages the trigeminal system through specific chemical binding. For instance, many gums or confectionaries incorporate chemical cooling agents, such as Wilkinson Sword (WS) compounds, to create the sensation of coldness. The current study was designed to evaluate crossmodal associations of color and aroma with the chemesthetic perception of cooling. A “minty” and non-odorized set of confectionary stimuli, colored green, blue or white, with moderate cooling properties (with WS-3) were used in this study. In the first session, participants were randomly presented a stimuli and asked to rate several attributes including its cooling intensity on a generalized Labeled Magnitude Scale (gLMS). In the second session, the same participants were asked to relate cooling levels to different colors and which color relates to the “minty” odor. Additionally, open-ended reasons were given for association choices. Appearance and odor influenced the intensity of cooling sensation. In particular, the odorized and blue samples were rated as cooler than the non-odorized and other colored samples, respectively. The follow-up session confirms blue as a color associated with cooling properties, especially cool objects/abstract concepts. Meanwhile, odor’s enhancement on cooling sensation may be more perceptual in nature through affective matching from enhanced flavor.

On the Relative Nature of (Pitch-Based) Crossmodal Correspondences

This review deals with the question of the relative vs absolute nature of crossmodal correspondences, with a specific focus on those correspondences involving the auditory dimension of pitch. Crossmodal correspondences have been defined as the often-surprising crossmodal associations that people experience between features, attributes, or dimensions of experience in different sensory modalities, when either physically present, or else merely imagined. In the literature, crossmodal correspondences have often been contrasted with synaesthesia in that the former are frequently said to be relative phenomena (e.g., it is the higher-pitched of two sounds that is matched with the smaller of two visual stimuli, say, rather than there being a specific one-to-one crossmodal mapping between a particular pitch of sound and size of object). By contrast, in the case of synaesthesia, the idiosyncratic mapping between inducer and concurrent tends to be absolute (e.g., it is a particular sonic inducer that elicits a specific colour concurrent). However, a closer analysis of the literature soon reveals that the distinction between relative and absolute in the case of crossmodal correspondences may not be as clear-cut as some commentators would have us believe. Furthermore, it is important to note that the relative vs absolute question may receive different answers depending on the particular (class of) correspondence under empirical investigation.

Synaesthesia

Synaesthesia is a curious anomaly of multisensory perception. When presented with stimulation in one sensory channel, in addition to the percept usually associated with that channel (inducer) a true synaesthetic experiences a second percept in another perceptual modality (concurrent). Although synaesthesia is not pathological, true synaesthetes are relatively rare and their synaesthetic associations tend to be quite idiosyncratic. For this reason, studying synaesthesia is difficult, but exciting new experimental results are beginning to clarify what makes the brain of synaesthetes special and the mechanisms that may produce the condition. Even more importantly, the related phenomenon known as ‘natural’ crossmodal associations is instead experienced by everyone, providing another useful domain for studying multisensory interactions with important implications for understanding our preferences for products in terms of spontaneously evoked associations, as well as for choosing appropriate names, labels, and packaging in marketing applications.