scholarly journals Two-Level Multimodal Fusion for Sentiment Analysis in Public Security

2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Jianguo Sun ◽  
Hanqi Yin ◽  
Ye Tian ◽  
Junpeng Wu ◽  
Linshan Shen ◽  
...  
Keyword(s):  
Sentiment Analysis ◽  
Multimodal Fusion ◽  
Public Security ◽  
Multimodal Data ◽  
Decision Level ◽  
Analysis Task ◽  
Fusion Methods ◽  
Decision Level Fusion ◽  
Level Fusion ◽  
Data Level

Large amounts of data are widely stored in cyberspace. Not only can they bring much convenience to people’s lives and work, but they can also assist the work in the information security field, such as microexpression recognition and sentiment analysis in the criminal investigation. Thus, it is of great significance to recognize and analyze the sentiment information, which is usually described by different modalities. Due to the correlation among different modalities data, multimodal can provide more comprehensive and robust information than unimodal in data analysis tasks. The complementary information from different modalities can be obtained by multimodal fusion methods. These approaches can process multimodal data through fusion algorithms and ensure the accuracy of the information used for subsequent classification or prediction tasks. In this study, a two-level multimodal fusion (TlMF) method with both data-level and decision-level fusion is proposed to achieve the sentiment analysis task. In the data-level fusion stage, a tensor fusion network is utilized to obtain the text-audio and text-video embeddings by fusing the text with audio and video features, respectively. During the decision-level fusion stage, the soft fusion method is adopted to fuse the classification or prediction results of the upstream classifiers, so that the final classification or prediction results can be as accurate as possible. The proposed method is tested on the CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets, and the empirical results and ablation studies confirm the effectiveness of TlMF in capturing useful information from all the test modalities.

scholarly journals Trimodal Attention Module for Multimodal Sentiment Analysis (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13803-13804
Author(s):  
Anirudh Bindiganavale Harish ◽  
Fatiha Sadat
Keyword(s):  
Sentiment Analysis ◽  
State Of The Art ◽  
Contextual Information ◽  
Multimodal Fusion ◽  
Decision Level ◽  
Training Approach ◽  
Decision Level Fusion ◽  
Multimodal Sentiment Analysis ◽  
The Individual ◽  
Level Fusion

In our research, we propose a new multimodal fusion architecture for the task of sentiment analysis. The 3 modalities used in this paper are text, audio and video. Most of the current methods deal with either a feature level or a decision level fusion. In contrast, we propose an attention-based deep neural network and a training approach to facilitate both feature and decision level fusion. Our network effectively leverages information across all three modalities using a 2 stage fusion process. We test our network on the individual utterance based contextual information extracted from the CMU-MOSI Dataset. A comparison is drawn between the state-of-the-art and our network.

scholarly journals Vision-Based Tunnel Lining Health Monitoring via Bi-Temporal Image Comparison and Decision-Level Fusion of Change Maps

Sensors ◽  
2021 ◽  
Vol 21 (12) ◽  
pp. 4040
Author(s):  
Leanne Attard ◽  
Carl James Debono ◽  
Gianluca Valentino ◽  
Mario Di Castro
Keyword(s):  
Change Detection ◽  
Health Monitoring ◽  
Detection System ◽  
Automatic Monitoring ◽  
Structural Health ◽  
Decision Level ◽  
Fusion Methods ◽  
Inspection Systems ◽  
Decision Level Fusion ◽  
Level Fusion

Tunnel structural health inspections are predominantly done through periodic visual observations, requiring humans to be physically present on-site, possibly exposing them to hazardous environments. These surveys are subjective (relying on the surveyor experience), time-consuming, and may demand operation shutdown. These issues can be mitigated through accurate automatic monitoring and inspection systems. In this work, we propose a remotely operated machine vision change detection application to improve the structural health monitoring of tunnels. The vision-based sensing system acquires the data from a rig of cameras hosted on a robotic platform that is driven parallel to the tunnel walls. These data are then pre-processed using image processing and deep learning techniques to reduce nuisance changes caused by light variations. Image fusion techniques are then applied to identify the changes occurring in the tunnel structure. Different pixel-based change detection approaches are used to generate temporal change maps. Decision-level fusion methods are then used to combine these change maps to obtain a more reliable detection of the changes that occur between surveys. A quantitative analysis of the results achieved shows that the proposed change detection system achieved a recall value of 81%, a precision value of 93% and an F1-score of 86.7%.

scholarly journals Feature and Decision Level Fusion in Children Multimodal Biometrics

2020 ◽  
Vol 8 (5) ◽  
pp. 2522-2527
Keyword(s):  
Nearest Neighbor ◽  
Principal Component ◽  
Identification Accuracy ◽  
Support Vector ◽  
Biometric System ◽  
K Nearest Neighbor ◽  
Decision Level ◽  
Multimodal Biometric System ◽  
Decision Level Fusion ◽  
Level Fusion

In this paper, we design method for recognition of fingerprint and IRIS using feature level fusion and decision level fusion in Children multimodal biometric system. Initially, Histogram of Gradients (HOG), Gabour and Maximum filter response are extracted from both the domains of fingerprint and IRIS and considered for identification accuracy. The combination of feature vector of all the possible features is recommended by biometrics traits of fusion. For fusion vector the Principal Component Analysis (PCA) is used to select features. The reduced features are fed into fusion classifier of K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Navie Bayes(NB). For children multimodal biometric system the suitable combination of features and fusion classifiers is identified. The experimentation conducted on children’s fingerprint and IRIS database and results reveal that fusion combination outperforms individual. In addition the proposed model advances the unimodal biometrics system.

Sign in / Sign up

Export Citation Format

Share Document