Information Fusion for Human Action Recognition via Biset/Multiset Globality Locality Preserving Canonical Correlation Analysis

2018 ◽  
Vol 27 (11) ◽  
pp. 5275-5287 ◽  
Author(s):  
Nour El Din Elmadany ◽  
Yifeng He ◽  
Ling Guan
Keyword(s):  
Correlation Analysis ◽  
Action Recognition ◽  
Canonical Correlation Analysis ◽  
Information Fusion ◽  
Canonical Correlation ◽  
Human Action Recognition ◽  
Human Action ◽  
Locality Preserving

scholarly journals Multimodal Information Fusion for Human Action Recognition

2021 ◽  
Author(s):  
Nour Elmadany
Keyword(s):  
Deep Learning ◽  
Correlation Analysis ◽  
Action Recognition ◽  
Canonical Correlation Analysis ◽  
Canonical Correlation ◽  
Human Action Recognition ◽  
Human Action ◽  
Local Network ◽  
Accelerometer Data ◽  
Fusion Technique

This thesis presents three frameworks of human action recognition to facilitate better recognition performance. The first framework fuses handcrafted features from four different modalities including RGB, depth, skeleton, and accelerometer data. In addition, a new descriptor for skeleton data is proposed that provides a discriminative representation for the poses of an action. Since the goal of the first framework is to find a more discriminative subspace, a generalized fusion technique Multimodal Hybrid Centroid Canonical Correlation Analysis (MHCCCA) is proposed for two or more sets of features or modalities. The second framework fuses handcrafted and deep learning features from three modalities including RGB, depth, and skeleton. In this framework a new depth representation is introduced that extracts the final representation using Deep ConvNet. The proposed fusion technique forms the backbone of the framework: Multiset Globality Locality Preserving Canonical Correlation Analysis (MGLPCCA) for two or more sets of features or modalities. MGLPCCA aims to preserve the local and global structures of data while maximizing the correlation among different modalities or sets. The third framework uses the deep learning techniques to improve the long term temporal modelling through two proposed techniques: Temporal Relational Network (TRN) and Temporal Second Order Pooling Based Network (T-SOPN). Additionally, Global-Local Network (GLN) and Fuse-Inception Network (FIN) are proposed to encourage the network to learn complementary information about the action and scene itself. Qualitative and quantitative experiments are conducted on nine different datasets demonstrating the effectiveness of the proposed framework over state-of-the-art methods.

scholarly journals Multimodal Information Fusion for Human Action Recognition

2021 ◽  
Author(s):  
Nour Elmadany
Keyword(s):  
Deep Learning ◽  
Correlation Analysis ◽  
Action Recognition ◽  
Canonical Correlation Analysis ◽  
Canonical Correlation ◽  
Human Action Recognition ◽  
Human Action ◽  
Local Network ◽  
Accelerometer Data ◽  
Fusion Technique

This thesis presents three frameworks of human action recognition to facilitate better recognition performance. The first framework fuses handcrafted features from four different modalities including RGB, depth, skeleton, and accelerometer data. In addition, a new descriptor for skeleton data is proposed that provides a discriminative representation for the poses of an action. Since the goal of the first framework is to find a more discriminative subspace, a generalized fusion technique Multimodal Hybrid Centroid Canonical Correlation Analysis (MHCCCA) is proposed for two or more sets of features or modalities. The second framework fuses handcrafted and deep learning features from three modalities including RGB, depth, and skeleton. In this framework a new depth representation is introduced that extracts the final representation using Deep ConvNet. The proposed fusion technique forms the backbone of the framework: Multiset Globality Locality Preserving Canonical Correlation Analysis (MGLPCCA) for two or more sets of features or modalities. MGLPCCA aims to preserve the local and global structures of data while maximizing the correlation among different modalities or sets. The third framework uses the deep learning techniques to improve the long term temporal modelling through two proposed techniques: Temporal Relational Network (TRN) and Temporal Second Order Pooling Based Network (T-SOPN). Additionally, Global-Local Network (GLN) and Fuse-Inception Network (FIN) are proposed to encourage the network to learn complementary information about the action and scene itself. Qualitative and quantitative experiments are conducted on nine different datasets demonstrating the effectiveness of the proposed framework over state-of-the-art methods.

Sign in / Sign up

Export Citation Format

Share Document