scholarly journals Multimodal feature fusion based on object relation for video captioning

2022 ◽  
Author(s):  
Zhiwen Yan ◽  
Ying Chen ◽  
Jinlong Song ◽  
Jia Zhu
Keyword(s):  
Feature Fusion ◽  
Object Relation ◽  
Video Captioning

Video Captioning with External Knowledge Assistance and Multi-feature Fusion

2021 ◽  
pp. 12-19
Author(s):  
Jiao-Wei Miao ◽  
Huan Shao ◽  
Yi Ji ◽  
Ying Li ◽  
Chun-Ping Liu
Keyword(s):  
Feature Fusion ◽  
Video Captioning ◽  
External Knowledge

A Study on Utilization of Three-Dimensional Sensor Lip Image for Developing a Pronunciation Recognition System

2019 ◽  
Vol 63 (5) ◽  
pp. 50402-1-50402-9 ◽  
Author(s):  
Ing-Jr Ding ◽  
Chong-Min Ruan
Keyword(s):  
Principal Component Analysis ◽  
Automatic Speech Recognition ◽  
Feature Fusion ◽  
Three Dimensional ◽  
Principal Component ◽  
Recognition System ◽  
Geometrical Characteristics ◽  
3D Geometry ◽  
Different Types ◽  
The Disabled

Abstract The acoustic-based automatic speech recognition (ASR) technique has been a matured technique and widely seen to be used in numerous applications. However, acoustic-based ASR will not maintain a standard performance for the disabled group with an abnormal face, that is atypical eye or mouth geometrical characteristics. For governing this problem, this article develops a three-dimensional (3D) sensor lip image based pronunciation recognition system where the 3D sensor is efficiently used to acquire the action variations of the lip shapes of the pronunciation action from a speaker. In this work, two different types of 3D lip features for pronunciation recognition are presented, 3D-(x, y, z) coordinate lip feature and 3D geometry lip feature parameters. For the 3D-(x, y, z) coordinate lip feature design, 18 location points, each of which has 3D-sized coordinates, around the outer and inner lips are properly defined. In the design of 3D geometry lip features, eight types of features considering the geometrical space characteristics of the inner lip are developed. In addition, feature fusion to combine both 3D-(x, y, z) coordinate and 3D geometry lip features is further considered. The presented 3D sensor lip image based feature evaluated the performance and effectiveness using the principal component analysis based classification calculation approach. Experimental results on pronunciation recognition of two different datasets, Mandarin syllables and Mandarin phrases, demonstrate the competitive performance of the presented 3D sensor lip image based pronunciation recognition system.

Sign in / Sign up

Export Citation Format

Share Document