Representation of inflected signs from American Sign Language in short-term memory

Individuals with hearing hindrance utilize gesture based communication to exchange their thoughts. Generally hand movements are used by them to communicate among themselves. But there are certain limitations when they communicate with other people who cannot understand these hand movements. There is a need to have a mechanism that can act as a translator between these people to communicate. It would be easier for these people to interact if there exists direct infrastructure that is able to convert signs to text and voice messages. As of late, numerous such frameworks for gesture based communication acknowledgment have been developed. But most of them are made either for static gesture recognition or dynamic gesture recognition. As sentences are generated using combinations of static and dynamic gestures, it would be simpler for hearing debilitated individuals if such computerized frameworks can detect both the static and dynamic motions together. We have proposed a design and architecture of American Sign Language (ASL) recognition with convolutional neural networks (CNN). This paper utilizes a pretrained VGG-16 architecture for static gesture recognition and for dynamic gesture recognition, spatiotemporal features were learnt with the complex architecture, called deep learning. It contains a bidirectional convolutional Long Short Term Memory network (ConvLSTM) and 3D convolutional neural network (3DCNN) and this architecture is responsible to extract 2D spatio temporal features.

Download Full-text

Recognition of Sign Language System for Indonesian Language Using Long Short-Term Memory Neural Networks

Advanced Science Letters ◽

10.1166/asl.2018.10675 ◽

2018 ◽

Vol 24 (2) ◽

pp. 999-1004 ◽

Cited By ~ 4

Author(s):

Erdefi Rakun ◽

Aniati M Arymurthy ◽

Lim Y Stefanus ◽

Alfan F Wicaksono ◽

I. Wayan W Wisesa

Keyword(s):

Neural Networks ◽

Sign Language ◽

Short Term Memory ◽

Short Term ◽

Term Memory ◽

Language System ◽

Long Short Term Memory

Download Full-text

Short-term recall by deaf signers of American Sign Language: Implications of encoding strategy for order recall.

Journal of Experimental Psychology Learning Memory and Cognition ◽

10.1037/0278-7393.8.6.572 ◽

1982 ◽

Vol 8 (6) ◽

pp. 572-583 ◽

Cited By ~ 45

Author(s):

Vicki L. Hanson

Keyword(s):

American Sign Language ◽

Sign Language ◽

American Sign ◽

Short Term ◽

Encoding Strategy ◽

Deaf Signers

Download Full-text

Neural Organization of Linguistic Short-term Memory is Sensory Modality–dependent: Evidence from Signed and Spoken Language

Journal of Cognitive Neuroscience ◽

10.1162/jocn.2008.20154 ◽

2008 ◽

Vol 20 (12) ◽

pp. 2198-2210 ◽

Cited By ~ 26

Author(s):

Judy Pa ◽

Stephen M. Wilson ◽

Herbert Pickell ◽

Ursula Bellugi ◽

Gregory Hickok

Keyword(s):

Neural Networks ◽

American Sign Language ◽

Short Term Memory ◽

Sensory Modality ◽

Maintenance Phase ◽

Sensory Stimulation ◽

Short Term ◽

Term Memory ◽

Neural Organization ◽

Signed Language

Despite decades of research, there is still disagreement regarding the nature of the information that is maintained in linguistic short-term memory (STM). Some authors argue for abstract phonological codes, whereas others argue for more general sensory traces. We assess these possibilities by investigating linguistic STM in two distinct sensory–motor modalities, spoken and signed language. Hearing bilingual participants (native in English and American Sign Language) performed equivalent STM tasks in both languages during functional magnetic resonance imaging. Distinct, sensory-specific activations were seen during the maintenance phase of the task for spoken versus signed language. These regions have been previously shown to respond to nonlinguistic sensory stimulation, suggesting that linguistic STM tasks recruit sensory-specific networks. However, maintenance-phase activations common to the two languages were also observed, implying some form of common process. We conclude that linguistic STM involves sensory-dependent neural networks, but suggest that sensory-independent neural networks may also exist.

Download Full-text

Spelling Correction Real-Time American Sign Language Alphabet Translation System Based on YOLO Network and LSTM

Electronics ◽

10.3390/electronics10091035 ◽

2021 ◽

Vol 10 (9) ◽

pp. 1035

Author(s):

Miguel Rivera-Acosta ◽

Juan Manuel Ruiz-Varela ◽

Susana Ortega-Cisneros ◽

Jorge Rivera ◽

Ramón Parra-Michel ◽

...

Keyword(s):

Neural Network ◽

Data Processing ◽

American Sign Language ◽

Real Time ◽

Sign Language ◽

Short Term Memory ◽

Translation System ◽

American Sign ◽

Spelling Correction ◽

Unseen Data

In this paper, we present a novel approach that aims to solve one of the main challenges in hand gesture recognition tasks in static images, to compensate for the accuracy lost when trained models are used to interpret completely unseen data. The model presented here consists of two main data-processing stages. A deep neural network (DNN) for performing handshape segmentation and classification is used in which multiple architectures and input image sizes were tested and compared to derive the best model in terms of accuracy and processing time. For the experiments presented in this work, the DNN models were trained with 24,000 images of 24 signs from the American Sign Language alphabet and fine-tuned with 5200 images of 26 generated signs. The system was real-time tested with a community of 10 persons, yielding a mean average precision and processing rate of 81.74% and 61.35 frames-per-second, respectively. As a second data-processing stage, a bidirectional long short-term memory neural network was implemented and analyzed for adding spelling correction capability to our system, which scored a training accuracy of 98.07% with a dictionary of 370 words, thus, increasing the robustness in completely unseen data, as shown in our experiments.

Download Full-text