scholarly journals Robust Hand Shape Features for Dynamic Hand Gesture Recognition Using Multi-Level Feature LSTM

2020 ◽  
Vol 10 (18) ◽  
pp. 6293
Author(s):  
Nhu-Tai Do ◽  
Soo-Hyung Kim ◽  
Hyung-Jeong Yang ◽  
Guee-Sang Lee
Keyword(s):  
Gesture Recognition ◽  
Point Cloud ◽  
Hand Gesture Recognition ◽  
Feature Representation ◽  
Geometric Transformation ◽  
Hand Gesture ◽  
Shape Features ◽  
Hand Shape ◽  
Dynamic Hand Gesture Recognition ◽  
Multi Level

This study builds robust hand shape features from the two modalities of depth and skeletal data for the dynamic hand gesture recognition problem. For the hand skeleton shape approach, we use the movement, the rotations of the hand joints with respect to their neighbors, and the skeletal point-cloud to learn the 3D geometric transformation. For the hand depth shape approach, we use the feature representation from the hand component segmentation model. Finally, we propose a multi-level feature LSTM with Conv1D, the Conv2D pyramid, and the LSTM block to deal with the diversity of hand features. Therefore, we propose a novel method by exploiting robust skeletal point-cloud features from skeletal data, as well as depth shape features from the hand component segmentation model in order for the multi-level feature LSTM model to benefit from both. Our proposed method achieves the best result on the Dynamic Hand Gesture Recognition (DHG) dataset with 14 and 28 classes for both depth and skeletal data with accuracies of 96.07% and 94.40%, respectively.

scholarly journals Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition

2019 ◽  
Vol 2019 (1) ◽  
Author(s):  
Yong Li ◽  
Zihang He ◽  
Xiang Ye ◽  
Zuguo He ◽  
Kangrong Han
Keyword(s):  
Gesture Recognition ◽  
Hand Gesture Recognition ◽  
Feature Representation ◽  
Training Dataset ◽  
Hand Gesture ◽  
Semantic Features ◽  
Convolutional Network ◽  
Convolutional Networks ◽  
Relative Coordinates ◽  
Dynamic Hand Gesture Recognition

Abstract Hand gesture recognition methods play an important role in human-computer interaction. Among these methods are skeleton-based recognition techniques that seem to be promising. In literature, several methods have been proposed to recognize hand gestures with skeletons. One problem with these methods is that they consider little the connectivity between the joints of a skeleton, constructing simple graphs for skeleton connectivity. Observing this, we built a new model of hand skeletons by adding three types of edges in the graph to finely describe the linkage action of joints. Then, an end-to-end deep neural network, hand gesture graph convolutional network, is presented in which the convolution is conducted only on linked skeleton joints. Since the training dataset is relatively small, this work proposes expanding the coordinate dimensionality so as to let models learn more semantic features. Furthermore, relative coordinates are employed to help hand gesture graph convolutional network learn the feature representation independent of the random starting positions of actions. The proposed method is validated on two challenging datasets, and the experimental results show that it outperforms the state-of-the-art methods. Furthermore, it is relatively lightweight in practice for hand skeleton-based gesture recognition.

Connectionist Temporal Classification Model for Dynamic Hand Gesture Recognition using RGB and Optical flow Data

2020 ◽  
Vol 17 (4) ◽  
pp. 497-506
Author(s):  
Sunil Patel ◽  
Ramji Makwana
Keyword(s):  
Neural Network ◽  
Optical Flow ◽  
Gesture Recognition ◽  
Hand Gesture Recognition ◽  
Classification Model ◽  
Hand Gesture ◽  
Flow Data ◽  
Dynamic Hand Gesture Recognition ◽  
Connectionist Temporal Classification

Automatic classification of dynamic hand gesture is challenging due to the large diversity in a different class of gesture, Low resolution, and it is performed by finger. Due to a number of challenges many researchers focus on this area. Recently deep neural network can be used for implicit feature extraction and Soft Max layer is used for classification. In this paper, we propose a method based on a two-dimensional convolutional neural network that performs detection and classification of hand gesture simultaneously from multimodal Red, Green, Blue, Depth (RGBD) and Optical flow Data and passes this feature to Long-Short Term Memory (LSTM) recurrent network for frame-to-frame probability generation with Connectionist Temporal Classification (CTC) network for loss calculation. We have calculated an optical flow from Red, Green, Blue (RGB) data for getting proper motion information present in the video. CTC model is used to efficiently evaluate all possible alignment of hand gesture via dynamic programming and check consistency via frame-to-frame for the visual similarity of hand gesture in the unsegmented input stream. CTC network finds the most probable sequence of a frame for a class of gesture. The frame with the highest probability value is selected from the CTC network by max decoding. This entire CTC network is trained end-to-end with calculating CTC loss for recognition of the gesture. We have used challenging Vision for Intelligent Vehicles and Applications (VIVA) dataset for dynamic hand gesture recognition captured with RGB and Depth data. On this VIVA dataset, our proposed hand gesture recognition technique outperforms competing state-of-the-art algorithms and gets an accuracy of 86%

Sign in / Sign up

Export Citation Format

Share Document