scholarly journals Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion

Sensors ◽  
2020 ◽  
Vol 20 (7) ◽  
pp. 2146
Author(s):  
Yuya Moroto ◽  
Keisuke Maeda ◽  
Takahiro Ogawa ◽  
Miki Haseyama
Keyword(s):  
Neural Network ◽  
Visual Attention ◽  
Feature Fusion ◽  
State Of The Art ◽  
Eye Gaze ◽  
Image Representation ◽  
Emotion Classification ◽  
Emotional Category ◽  
Tensor Regression ◽  
The Relationship

The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified.

scholarly journals Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network

2017 ◽  
Author(s):  
Jufeng Yang ◽  
Dongyu She ◽  
Ming Sun
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Prior Knowledge ◽  
State Of The Art ◽  
Ground Truth ◽  
Majority Voting ◽  
Emotion Classification ◽  
Voting Scheme ◽  
Label Distribution ◽  
Similarity Information

Visual sentiment analysis is attracting more and more attention with the increasing tendency to express emotions through visual contents. Recent algorithms in convolutional neural networks (CNNs) considerably advance the emotion classification, which aims to distinguish differences among emotional categories and assigns a single dominant label to each image. However, the task is inherently ambiguous since an image usually evokes multiple emotions and its annotation varies from person to person. In this work, we address the problem via label distribution learning (LDL) and develop a multi-task deep framework by jointly optimizing both classification and distribution prediction. While the proposed method prefers to the distribution dataset with annotations of different voters, the majority voting scheme is widely adopted as the ground truth in this area, and few dataset has provided multiple affective labels. Hence, we further exploit two weak forms of prior knowledge, which are expressed as similarity information between labels, to generate emotional distribution for each category. The experiments conducted on both distribution datasets, i.e., Emotion6, Flickr_LDL, Twitter_LDL, and the largest single emotion dataset, i.e., Flickr and Instagram, demonstrate the proposed method outperforms the state-of-the-art approaches.

scholarly journals Multilevel Attention Residual Neural Network for Multimodal Online Social Network Rumor Detection

2021 ◽  
Vol 9 ◽  
Author(s):  
Zhuang Wang ◽  
Jie Sui
Keyword(s):  
Neural Network ◽  
Social Network ◽  
State Of The Art ◽  
Online Social Network ◽  
Image Representation ◽  
Rapid Rise ◽  
Text And Image ◽  
Detection Model ◽  
Fused Image ◽  
Rumor Detection

In recent years, with the rapid rise of social networks, such as Weibo and Twitter, multimodal social network rumors have also spread. Unlike traditional unimodal rumor detection, the main difficulty of multimodal rumor detection is in avoiding the generation of noise information while using the complementarity of different modal features. In this article, we propose a multimodal online social network rumor detection model based on the multilevel attention residual neural network (MARN). First, the features of text and image are extracted by Bert and ResNet-18, respectively, and the cross-attention residual mechanism is used to enhance the representation of images with a text vector. Second, the enhanced image vector and text vector are concatenated and fused by the self-attention residual mechanism. Finally, the fused image–text vectors are classified into two categories. Among them, the attention mechanism can effectively enhance the image representation and further improve the fusion effect between the image and the text, while the residual mechanism retains the unique attributes of each original modal feature while using different modal features. To assess the performance of the MARN model, we conduct experiments on the Weibo dataset, and the results show that the MARN model outperforms the state-of-the-art models in terms of accuracy and F1 value.

scholarly journals Pedestrian Attribute Recognition with Graph Convolutional Network in Surveillance Scenarios

2019 ◽  
Vol 11 (11) ◽  
pp. 245 ◽  
Author(s):  
Xiangpeng Song ◽  
Hongbin Yang ◽  
Congcong Zhou
Keyword(s):  
Neural Network ◽  
Spatial Distribution ◽  
Correlation Matrix ◽  
State Of The Art ◽  
Recognition Performance ◽  
Image Representation ◽  
Normal Operation ◽  
Convolutional Network ◽  
Attribute Recognition ◽  
Novel Model

Pedestrian attribute recognition is to predict a set of attribute labels of the pedestrian from surveillance scenarios, which is a very challenging task for computer vision due to poor image quality, continual appearance variations, as well as diverse spatial distribution of imbalanced attributes. It is desirable to model the label dependencies between different attributes to improve the recognition performance as each pedestrian normally possesses many attributes. In this paper, we treat pedestrian attribute recognition as multi-label classification and propose a novel model based on the graph convolutional network (GCN). The model is mainly divided into two parts, we first use convolutional neural network (CNN) to extract pedestrian feature, which is a normal operation processing image in deep learning, then we transfer attribute labels to word embedding and construct a correlation matrix between labels to help GCN propagate information between nodes. This paper applies the object classifiers learned by GCN to the image representation extracted by CNN to enable the model to have the ability to be end-to-end trainable. Experiments on pedestrian attribute recognition dataset show that the approach obviously outperforms other existing state-of-the-art methods.

scholarly journals Action Recognition Based on the Fusion of Graph Convolutional Networks with High Order Features

2020 ◽  
Vol 10 (4) ◽  
pp. 1482 ◽  
Author(s):  
Jiuqing Dong ◽  
Yongbin Gao ◽  
Hyo Jong Lee ◽  
Heng Zhou ◽  
Yifan Yao ◽  
...  
Keyword(s):  
Neural Network ◽  
Action Recognition ◽  
Feature Fusion ◽  
State Of The Art ◽  
Recognition Task ◽  
High Order ◽  
Recognition Method ◽  
Related Research ◽  
Convolutional Networks ◽  
Temporal Features

Skeleton-based action recognition is a widely used task in action related research because of its clear features and the invariance of human appearances and illumination. Furthermore, it can also effectively improve the robustness of the action recognition. Graph convolutional networks have been implemented on those skeletal data to recognize actions. Recent studies have shown that the graph convolutional neural network works well in the action recognition task using spatial and temporal features of skeleton data. The prevalent methods to extract the spatial and temporal features purely rely on a deep network to learn from primitive 3D position. In this paper, we propose a novel action recognition method applying high-order spatial and temporal features from skeleton data, such as velocity features, acceleration features, and relative distance between 3D joints. Meanwhile, a method of multi-stream feature fusion is adopted to fuse these high-order features we proposed. Extensive experiments on Two large and challenging datasets, NTU-RGBD and NTU-RGBD-120, indicate that our model achieves the state-of-the-art performance.

scholarly journals Label-Attended Hashing for Multi-Label Image Retrieval

2020 ◽  
Author(s):  
Yanzhao Xie ◽  
Yu Liu ◽  
Yangtao Wang ◽  
Lianli Gao ◽  
Peng Wang ◽  
...  
Keyword(s):  
Neural Network ◽  
Feature Extraction ◽  
Image Retrieval ◽  
Hash Function ◽  
State Of The Art ◽  
Image Representation ◽  
Back Propagation ◽  
Cauchy Distribution ◽  
Representation Learning ◽  
Hash Codes

For the multi-label image retrieval, the existing hashing algorithms neglect the dependency between objects and thus fail to capture the attention information in the feature extraction, which affects the precision of hash codes. To address this problem, we explore the inter-dependency between objects through their co-occurrence correlation from the label set and adopt Multi-modal Factorized Bilinear (MFB) pooling component so that the image representation learning can capture this attention information. We propose a Label-Attended Hashing (LAH) algorithm which enables an end-to-end hash model with inter-dependency feature extraction. LAH first combines Convolutional Neural Network (CNN) and Graph Convolution Network (GCN) to separately generate the image representation and label co-occurrence embeddings, then adopts MFB to fuse these two modal vectors, finally learns the hash function with a Cauchy distribution based loss function via back propagation. Extensive experiments on public multi-label datasets demonstrate that (1) LAH can achieve the state-of-the-art retrieval results and (2) the usage of co-occurrence relationship and MFB not only promotes the precision of hash codes but also accelerates the hash learning. GitHub address: https://github.com/IDSM-AI/LAH.

scholarly journals Hierarchical Attention Network with Pairwise Loss for Chinese Zero Pronoun Resolution

2020 ◽  
Vol 34 (05) ◽  
pp. 8352-8359
Author(s):  
Peiqin Lin ◽  
Meng Yang
Keyword(s):  
Neural Network ◽  
State Of The Art ◽  
Classification Task ◽  
Pronoun Resolution ◽  
Attention Network ◽  
Attention Model ◽  
Network Methods ◽  
Zero Pronouns ◽  
The Relationship ◽  
Zero Pronoun

Recent neural network methods for Chinese zero pronoun resolution didn't take bidirectional attention between zero pronouns and candidate antecedents into consideration, and simply treated the task as a classification task, ignoring the relationship between different candidates of a zero pronoun. To solve these problems, we propose a Hierarchical Attention Network with Pairwise Loss (HAN-PL), for Chinese zero pronoun resolution. In the proposed HAN-PL, we design a two-layer attention model to generate more powerful representations for zero pronouns and candidate antecedents. Furthermore, we propose a novel pairwise loss by introducing the correct-antecedent similarity constraint and the pairwise-margin loss, making the learned model more discriminative. Extensive experiments have been conducted on OntoNotes 5.0 dataset, and our model achieves state-of-the-art performance in the task of Chinese zero pronoun resolution.

Sign in / Sign up

Export Citation Format

Share Document