Open-Domain Audio-Visual Speech Recognition: A Deep Learning Approach

2016 ◽  
Author(s):  
Yajie Miao ◽  
Florian Metze
Keyword(s):  
Deep Learning ◽  
Speech Recognition ◽  
Visual Speech ◽  
Learning Approach ◽  
Open Domain ◽  
Visual Speech Recognition

Resource-Adaptive Deep Learning for Visual Speech Recognition

2020 ◽  
Author(s):  
Alexandros Koumparoulis ◽  
Gerasimos Potamianos ◽  
Samuel Thomas ◽  
Edmilson da Silva Morais
Keyword(s):  
Deep Learning ◽  
Speech Recognition ◽  
Visual Speech ◽  
Visual Speech Recognition

scholarly journals Audio-visual speech recognition using deep learning

2014 ◽  
Vol 42 (4) ◽  
pp. 722-737 ◽  
Author(s):  
Kuniaki Noda ◽  
Yuki Yamaguchi ◽  
Kazuhiro Nakadai ◽  
Hiroshi G. Okuno ◽  
Tetsuya Ogata
Keyword(s):  
Deep Learning ◽  
Speech Recognition ◽  
Visual Speech ◽  
Visual Speech Recognition

scholarly journals Biosignal Sensors and Deep Learning-Based Speech Recognition: A Review

Sensors ◽  
2021 ◽  
Vol 21 (4) ◽  
pp. 1399
Author(s):  
Wookey Lee ◽  
Jessica Jiwon Seong ◽  
Busra Ozlu ◽  
Bong Sup Shim ◽  
Azizbek Marakhimov ◽  
...  
Keyword(s):  
Deep Learning ◽  
Speech Recognition ◽  
Learning Technologies ◽  
Magnetic Sensors ◽  
Visual Speech ◽  
Future Research ◽  
Electromagnetic Articulography ◽  
Speech Interface ◽  
Visual Speech Recognition ◽  
Being There

Voice is one of the essential mechanisms for communicating and expressing one’s intentions as a human being. There are several causes of voice inability, including disease, accident, vocal abuse, medical surgery, ageing, and environmental pollution, and the risk of voice loss continues to increase. Novel approaches should have been developed for speech recognition and production because that would seriously undermine the quality of life and sometimes leads to isolation from society. In this review, we survey mouth interface technologies which are mouth-mounted devices for speech recognition, production, and volitional control, and the corresponding research to develop artificial mouth technologies based on various sensors, including electromyography (EMG), electroencephalography (EEG), electropalatography (EPG), electromagnetic articulography (EMA), permanent magnet articulography (PMA), gyros, images and 3-axial magnetic sensors, especially with deep learning techniques. We especially research various deep learning technologies related to voice recognition, including visual speech recognition, silent speech interface, and analyze its flow, and systematize them into a taxonomy. Finally, we discuss methods to solve the communication problems of people with disabilities in speaking and future research with respect to deep learning components.

Audio-visual speech Recognition using LSTM and CNN

2019 ◽  
Vol 13 ◽  
Author(s):  
Eslam E. El Maghraby ◽  
Amr M. Gody ◽  
M. Hesham Farouk
Keyword(s):  
Deep Learning ◽  
Speech Recognition ◽  
Speech Signal ◽  
Recognition Accuracy ◽  
Recognition System ◽  
Visual Speech ◽  
Robust Speech Recognition ◽  
Noisy Environment ◽  
Visual Speech Recognition ◽  
Proposed Model

Background: Multimodal speech recognition is proved to be one of the most promising solutions for robust speech recognition, especially when the audio signal is corrupted by noise. As the visual speech signal not affected by audio noise, it can be used to obtain more information used to enhance the speech recognition accuracy in noisy system. The critical stage in designing robust speech recognition system is choosing of reliable classification method from large variety of available classification techniques. Deep learning is well-known as a technique that has the ability to classify a nonlinear problem, and takes into consideration the sequential characteristic of the speech signal. Numerous researches have been done in applying deep learning to overcome Audio-Visual Speech Recognition (AVSR) problems due to its amazing achievements in both speech and image recognition. Even though optimistic results have been obtained from the continuous studies, researches on enhancing accuracy in noise system and selecting the best classification technique are still gaining lots of attention. Objective: This paper aims to build AVSR system that uses both acoustic combined with visual speech information and use classification technique based on deep learning to improve the recognition performance in a clean and noisy environment. Method: Mel frequency cepstral coefficient (MFCC) and Discrete Cosine Transform (DCT) are used to extract the effective features from audio and visual speech signal respectively. The audio feature rate is greater than the visual feature rate, so that linear interpolation is needed to obtain equal feature vectors size then early integrating them to get combined feature vector. Bidirectional Long-Short Term Memory (BiLSTM), one of the Deep learning techniques, are used for classification process and compare the obtained results to other classification techniques like Convolution Neural Network (CNN) and the traditional Hidden Markov Models (HMM). The effectiveness of the proposed model is proved by using two multi-speaker AVSR datasets termed AV letters and GRID. Results: The proposed model gives promising results where the obtained results In case of GRID, using integrated audio-visual features achieved highest recognition accuracy of 99.07% and 98.47% , with enhancement up to 9.28% and 12.05% over audio-only for clean and noisy data respectively. For AVletters, the highest recognition accuracy is 93.33% with enhancement up to 8.33% over audio-only. Conclusion: Based on the obtained results, we can conclude that increasing the size of audio feature vector from 13 to 39 doesn’t give effective enhancement for the recognition accuracy in clean environment, but in noisy environment, it gives better performance. BiLSTM is considered to be the optimal classifier for a robust speech recognition system when compared to CNN and traditional HMM, because it takes into consideration the sequential characteristic of the speech signal (audio and visual). The proposed model gives great improvement in the recognition accuracy and decreasing the loss value for both clean and noisy environments than using audio-only features. Comparing the proposed model to previously obtain results which using the same datasets, we found that our model gives higher recognition accuracy and confirms the robustness of our model.

Asynchrony modeling for audio-visual speech recognition

2002 ◽  
Author(s):  
Guillaume Gravier ◽  
Gerasimos Potamianos ◽  
Chalapathy Neti
Keyword(s):  
Speech Recognition ◽  
Visual Speech ◽  
Visual Speech Recognition
Sign in / Sign up

Export Citation Format

Share Document