Boosting-Based Multimodal Speaker Detection for Distributed Meetings

Simulating Realistically-Spatialised Simultaneous Speech Using Video-Driven Speaker Detection and the CHiME-5 Dataset

10.21437/interspeech.2020-2807 ◽

2020 ◽

Author(s):

Jack Deadman ◽

Jon Barker

Keyword(s):

Speaker Detection

Download Full-text

Bio-Inspired Modality Fusion for Active Speaker Detection

Applied Sciences ◽

10.3390/app11083397 ◽

2021 ◽

Vol 11 (8) ◽

pp. 3397

Author(s):

Gustavo Assunção ◽

Nuno Gonçalves ◽

Paulo Menezes

Keyword(s):

Superior Colliculus ◽

Visual Information ◽

Human Beings ◽

Validation Process ◽

Detection Approach ◽

Wide Range ◽

Speaker Detection ◽

The One ◽

The Brain ◽

Fusion Ability

Human beings have developed fantastic abilities to integrate information from various sensory sources exploring their inherent complementarity. Perceptual capabilities are therefore heightened, enabling, for instance, the well-known "cocktail party" and McGurk effects, i.e., speech disambiguation from a panoply of sound signals. This fusion ability is also key in refining the perception of sound source location, as in distinguishing whose voice is being heard in a group conversation. Furthermore, neuroscience has successfully identified the superior colliculus region in the brain as the one responsible for this modality fusion, with a handful of biological models having been proposed to approach its underlying neurophysiological process. Deriving inspiration from one of these models, this paper presents a methodology for effectively fusing correlated auditory and visual information for active speaker detection. Such an ability can have a wide range of applications, from teleconferencing systems to social robotics. The detection approach initially routes auditory and visual information through two specialized neural network structures. The resulting embeddings are fused via a novel layer based on the superior colliculus, whose topological structure emulates spatial neuron cross-mapping of unimodal perceptual fields. The validation process employed two publicly available datasets, with achieved results confirming and greatly surpassing initial expectations.

Download Full-text