Visual Signals in the Mammalian Auditory System

Coordination between different sensory systems is a necessary element of sensory processing. Where and how signals from different sense organs converge onto common neural circuitry have become topics of increasing interest in recent years. In this article, we focus specifically on visual–auditory interactions in areas of the mammalian brain that are commonly considered to be auditory in function. The auditory cortex and inferior colliculus are two key points of entry where visual signals reach the auditory pathway, and both contain visual- and/or eye movement–related signals in humans and other animals. The visual signals observed in these auditory structures reflect a mixture of visual modulation of auditory-evoked activity and visually driven responses that are selective for stimulus location or features. These key response attributes also appear in the classic visual pathway but may play a different role in the auditory pathway: to modify auditory rather than visual perception. Finally, while this review focuses on two particular areas of the auditory pathway where this question has been studied, robust descending as well as ascending connections within this pathway suggest that undiscovered visual signals may be present at other stages as well. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

A Study on Persistence of GAN-Based Vision-Induced Gustatory Manipulation

Vision-induced gustatory manipulation interfaces can help people with dietary restrictions feel as if they are eating what they want by modulating the appearance of the alternative foods they are eating in reality. However, it is still unclear whether vision-induced gustatory change persists beyond a single bite, how the sensation changes over time, and how it varies among individuals from different cultural backgrounds. The present paper reports on a user study conducted to answer these questions using a generative adversarial network (GAN)-based real-time image-to-image translation system. In the user study, 16 participants were presented somen noodles or steamed rice through a video see-through head mounted display (HMD) both in two conditions; without or with visual modulation (somen noodles and steamed rice were translated into ramen noodles and curry and rice, respectively), and brought food to the mouth and tasted it five times with an interval of two minutes. The results of the experiments revealed that vision-induced gustatory manipulation is persistent in many participants. Their persistent gustatory changes are divided into three groups: those in which the intensity of the gustatory change gradually increased, those in which it gradually decreased, and those in which it did not fluctuate, each with about the same number of participants. Although the generalizability is limited due to the small population, it was also found that non-Japanese and male participants tended to perceive stronger gustatory manipulation compared to Japanese and female participants. We believe that our study deepens our understanding and insight into vision-induced gustatory manipulation and encourages further investigation.

Rhythm of music seen through dance: Probing music–dance coupling by audiovisual meter perception

Action–perception coupling in music has been evidenced not only by how patterns of human dance reflect the metrical structure of musical rhythm, but also that moving to music modulates rhythm perception. Given the inherent connection between music and dance, this research investigated whether dance observation could induce meter perception of the visual rhythm similarly to the musical counterpart, and whether the visual meter could modulate concurrent auditory metrical perception. In Experiment 1, participants watched a point-light figure dance to a rhythm they heard simultaneously, both of which varied in meter. Participants responded whether the dance matched the rhythm, and the results were consistent with the imposed audiovisual meter match / mismatch, suggesting that participants could extract the visual meter from dance and compare it to the auditory meter. In Experiment 2, participants watched the dance in different meters while listening to a metrically ambiguous rhythm, which they (to some extent) subsequently identified as being more similar to another rhythm accentuated in the same meter as the dance than one in a different meter. The data partially supported visual modulation of auditory metrical interpretation. Together these results demonstrate parallels in meter perception between music and dance, which may share a common action representation that mediates cross-modal interactions. The findings also support theories of embodied musical rhythm from the perspective of visual-motor simulation.