scholarly journals Experimenting the Automatic Recognition of Non-Conventionalized Units in Sign Language

Algorithms ◽  
2020 ◽  
Vol 13 (12) ◽  
pp. 310
Author(s):  
Valentin Belissen ◽  
Annelies Braffort ◽  
Michèle Gouiffès
Keyword(s):  
Neural Network ◽  
Parallel Processing ◽  
Convolutional Neural Network ◽  
Sign Language ◽  
Performance Metrics ◽  
Automatic Recognition ◽  
Upper Body ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Languages

Sign Languages (SLs) are visual–gestural languages that have developed naturally in deaf communities. They are based on the use of lexical signs, that is, conventionalized units, as well as highly iconic structures, i.e., when the form of an utterance and the meaning it carries are not independent. Although most research in automatic Sign Language Recognition (SLR) has focused on lexical signs, we wish to broaden this perspective and consider the recognition of non-conventionalized iconic and syntactic elements. We propose the use of corpora made by linguists like the finely and consistently annotated dialogue corpus Dicta-Sign-LSF-v2. We then redefined the problem of automatic SLR as the recognition of linguistic descriptors, with carefully thought out performance metrics. Moreover, we developed a compact and generalizable representation of signers in videos by parallel processing of the hands, face and upper body, then an adapted learning architecture based on a Recurrent Convolutional Neural Network (RCNN). Through a study focused on the recognition of four linguistic descriptors, we show the soundness of the proposed approach and pave the way for a wider understanding of Continuous Sign Language Recognition (CSLR).

scholarly journals Sign Language Recognition Using Two-Stream Convolutional Neural Networks with Wi-Fi Signals

2020 ◽  
Vol 10 (24) ◽  
pp. 9005
Author(s):  
Chien-Cheng Lee ◽  
Zhongjian Gao
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Sign Language ◽  
Recognition Accuracy ◽  
Signal Interference ◽  
Stream Networks ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Stream Network ◽  
Value Decomposition

Sign language is an important way for deaf people to understand and communicate with others. Many researchers use Wi-Fi signals to recognize hand and finger gestures in a non-invasive manner. However, Wi-Fi signals usually contain signal interference, background noise, and mixed multipath noise. In this study, Wi-Fi Channel State Information (CSI) is preprocessed by singular value decomposition (SVD) to obtain the essential signals. Sign language includes the positional relationship of gestures in space and the changes of actions over time. We propose a novel dual-output two-stream convolutional neural network. It not only combines the spatial-stream network and the motion-stream network, but also effectively alleviates the backpropagation problem of the two-stream convolutional neural network (CNN) and improves its recognition accuracy. After the two stream networks are fused, an attention mechanism is applied to select the important features learned by the two-stream networks. Our method has been validated by the public dataset SignFi and adopted five-fold cross-validation. Experimental results show that SVD preprocessing can improve the performance of our dual-output two-stream network. For home, lab, and lab + home environment, the average recognition accuracy rates are 99.13%, 96.79%, and 97.08%, respectively. Compared with other methods, our method has good performance and better generalization capability.

Sign in / Sign up

Export Citation Format

Share Document