scholarly journals Action Recognition Network Using Stacked Short-Term Deep Features and Bidirectional Moving Average

2021 ◽  
Vol 11 (12) ◽  
pp. 5563
Author(s):  
Jinsol Ha ◽  
Joongchol Shin ◽  
Hasil Park ◽  
Joonki Paik
Keyword(s):  
Action Recognition ◽  
High Speed ◽  
Moving Average ◽  
Video Clip ◽  
Visual Surveillance ◽  
Temporal Information ◽  
Feature Maps ◽  
Short Term ◽  
Accurate Analysis ◽  
Difference Images

Action recognition requires the accurate analysis of action elements in the form of a video clip and a properly ordered sequence of the elements. To solve the two sub-problems, it is necessary to learn both spatio-temporal information and the temporal relationship between different action elements. Existing convolutional neural network (CNN)-based action recognition methods have focused on learning only spatial or temporal information without considering the temporal relation between action elements. In this paper, we create short-term pixel-difference images from the input video, and take the difference images as an input to a bidirectional exponential moving average sub-network to analyze the action elements and their temporal relations. The proposed method consists of: (i) generation of RGB and differential images, (ii) extraction of deep feature maps using an image classification sub-network, (iii) weight assignment to extracted feature maps using a bidirectional, exponential, moving average sub-network, and (iv) late fusion with a three-dimensional convolutional (C3D) sub-network to improve the accuracy of action recognition. Experimental results show that the proposed method achieves a higher performance level than existing baseline methods. In addition, the proposed action recognition network takes only 0.075 seconds per action class, which guarantees various high-speed or real-time applications, such as abnormal action classification, human–computer interaction, and intelligent visual surveillance.

scholarly journals Hierarchical dynamic depth projected difference images–based action recognition in videos with convolutional neural networks

2019 ◽  
Vol 16 (1) ◽  
pp. 172988141882509 ◽  
Author(s):  
Hanbo Wu ◽  
Xin Ma ◽  
Yibin Li
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Action Recognition ◽  
Human Action Recognition ◽  
Human Action ◽  
Temporal Information ◽  
Superior Performance ◽  
Video Sequences ◽  
Depth Video ◽  
Difference Images

Temporal information plays a significant role in video-based human action recognition. How to effectively extract the spatial–temporal characteristics of actions in videos has always been a challenging problem. Most existing methods acquire spatial and temporal cues in videos individually. In this article, we propose a new effective representation for depth video sequences, called hierarchical dynamic depth projected difference images that can aggregate the action spatial and temporal information simultaneously at different temporal scales. We firstly project depth video sequences onto three orthogonal Cartesian views to capture the 3D shape and motion information of human actions. Hierarchical dynamic depth projected difference images are constructed with the rank pooling in each projected view to hierarchically encode the spatial–temporal motion dynamics in depth videos. Convolutional neural networks can automatically learn discriminative features from images and have been extended to video classification because of their superior performance. To verify the effectiveness of hierarchical dynamic depth projected difference images representation, we construct a hierarchical dynamic depth projected difference images–based action recognition framework where hierarchical dynamic depth projected difference images in three views are fed into three identical pretrained convolutional neural networks independently for finely retuning. We design three classification schemes in the framework and different schemes utilize different convolutional neural network layers to compare their effects on action recognition. Three views are combined to describe the actions more comprehensively in each classification scheme. The proposed framework is evaluated on three challenging public human action data sets. Experiments indicate that our method has better performance and can provide discriminative spatial–temporal information for human action recognition in depth videos.

scholarly journals Low-Cost Embedded System Using Convolutional Neural Networks-Based Spatiotemporal Feature Map for Real-Time Human Action Recognition

2021 ◽  
Vol 11 (11) ◽  
pp. 4940
Author(s):  
Jinsoo Kim ◽  
Jeongho Cho
Keyword(s):  
Embedded System ◽  
Real Time ◽  
Action Recognition ◽  
Processing Speed ◽  
Recognition Accuracy ◽  
Low Cost ◽  
Human Action Recognition ◽  
Human Action ◽  
Video Data ◽  
Feature Maps

The field of research related to video data has difficulty in extracting not only spatial but also temporal features and human action recognition (HAR) is a representative field of research that applies convolutional neural network (CNN) to video data. The performance for action recognition has improved, but owing to the complexity of the model, some still limitations to operation in real-time persist. Therefore, a lightweight CNN-based single-stream HAR model that can operate in real-time is proposed. The proposed model extracts spatial feature maps by applying CNN to the images that develop the video and uses the frame change rate of sequential images as time information. Spatial feature maps are weighted-averaged by frame change, transformed into spatiotemporal features, and input into multilayer perceptrons, which have a relatively lower complexity than other HAR models; thus, our method has high utility in a single embedded system connected to CCTV. The results of evaluating action recognition accuracy and data processing speed through challenging action recognition benchmark UCF-101 showed higher action recognition accuracy than the HAR model using long short-term memory with a small amount of video frames and confirmed the real-time operational possibility through fast data processing speed. In addition, the performance of the proposed weighted mean-based HAR model was verified by testing it in Jetson NANO to confirm the possibility of using it in low-cost GPU-based embedded systems.

scholarly journals Long- and Short-Term Conductance Control of Artificial Polymer Wire Synapses

Polymers ◽  
2021 ◽  
Vol 13 (2) ◽  
pp. 312
Author(s):  
Naruki Hagiwara ◽  
Shoma Sekizaki ◽  
Yuji Kuwahara ◽  
Tetsuya Asai ◽  
Megumi Akai-Kasaya
Keyword(s):  
Artificial Intelligence ◽  
Human Brain ◽  
High Speed ◽  
Conductive Polymer ◽  
Long Term Potentiation ◽  
Short Term ◽  
Synaptic Functions ◽  
Term Potentiation ◽  
Artificial Brain ◽  
External Stimuli

Networks in the human brain are extremely complex and sophisticated. The abstract model of the human brain has been used in software development, specifically in artificial intelligence. Despite the remarkable outcomes achieved using artificial intelligence, the approach consumes a huge amount of computational resources. A possible solution to this issue is the development of processing circuits that physically resemble an artificial brain, which can offer low-energy loss and high-speed processing. This study demonstrated the synaptic functions of conductive polymer wires linking arbitrary electrodes in solution. By controlling the conductance of the wires, synaptic functions such as long-term potentiation and short-term plasticity were achieved, which are similar to the manner in which a synapse changes the strength of its connections. This novel organic artificial synapse can be used to construct information-processing circuits by wiring from scratch and learning efficiently in response to external stimuli.

scholarly journals Application of Skeleton Data and Long Short-Term Memory in Action Recognition of Children with Autism Spectrum Disorder

Sensors ◽  
2021 ◽  
Vol 21 (2) ◽  
pp. 411
Author(s):  
Yunkai Zhang ◽  
Yinghong Tian ◽  
Pingyi Wu ◽  
Dongfan Chen
Keyword(s):  
Autism Spectrum Disorder ◽  
Action Recognition ◽  
Short Term Memory ◽  
Autism Spectrum ◽  
Spectrum Disorder ◽  
Short Term ◽  
Term Memory ◽  
Clinical Observations ◽  
The Neural Network ◽  
Long Short Term Memory

The recognition of stereotyped action is one of the core diagnostic criteria of Autism Spectrum Disorder (ASD). However, it mainly relies on parent interviews and clinical observations, which lead to a long diagnosis cycle and prevents the ASD children from timely treatment. To speed up the recognition process of stereotyped actions, a method based on skeleton data and Long Short-Term Memory (LSTM) is proposed in this paper. In the first stage of our method, the OpenPose algorithm is used to obtain the initial skeleton data from the video of ASD children. Furthermore, four denoising methods are proposed to eliminate the noise of the initial skeleton data. In the second stage, we track multiple ASD children in the same scene by matching distance between current skeletons and previous skeletons. In the last stage, the neural network based on LSTM is proposed to classify the ASD children’s actions. The performed experiments show that our proposed method is effective for ASD children’s action recognition. Compared to the previous traditional schemes, our scheme has higher accuracy and is almost non-invasive for ASD children.

scholarly journals Multiview deep learning based on tensor decomposition and its application in fault detection of overhead contact systems

2021 ◽  
Author(s):  
Xuewu Zhang ◽  
Yansheng Gong ◽  
Chen Qiao ◽  
Wenfeng Jing
Keyword(s):  
High Speed ◽  
Tensor Decomposition ◽  
Detection Methods ◽  
Detection Accuracy ◽  
Feature Maps ◽  
Training Time ◽  
Detection Model ◽  
Railway Line ◽  
Result Show ◽  
Deep Layers

AbstractThis article mainly focuses on the most common types of high-speed railways malfunctions in overhead contact systems, namely, unstressed droppers, foreign-body invasions, and pole number-plate malfunctions, to establish a deep-network detection model. By fusing the feature maps of the shallow and deep layers in the pretraining network, global and local features of the malfunction area are combined to enhance the network's ability of identifying small objects. Further, in order to share the fully connected layers of the pretraining network and reduce the complexity of the model, Tucker tensor decomposition is used to extract features from the fused-feature map. The operation greatly reduces training time. Through the detection of images collected on the Lanxin railway line, experiments result show that the proposed multiview Faster R-CNN based on tensor decomposition had lower miss probability and higher detection accuracy for the three types faults. Compared with object-detection methods YOLOv3, SSD, and the original Faster R-CNN, the average miss probability of the improved Faster R-CNN model in this paper is decreased by 37.83%, 51.27%, and 43.79%, respectively, and average detection accuracy is increased by 3.6%, 9.75%, and 5.9%, respectively.

scholarly journals Towards an Efficient CNN Inference Architecture Enabling In-Sensor Processing

Sensors ◽  
2021 ◽  
Vol 21 (6) ◽  
pp. 1955
Author(s):  
Md Jubaer Hossain Pantho ◽  
Pankaj Bhowmik ◽  
Christophe Bobda
Keyword(s):  
Power Consumption ◽  
High Speed ◽  
Image Sensor ◽  
Machine Learning Algorithms ◽  
Hierarchical Optimization ◽  
Computation Method ◽  
Optimization Approach ◽  
Efficient Computation ◽  
Feature Maps ◽  
Dynamic Power

The astounding development of optical sensing imaging technology, coupled with the impressive improvements in machine learning algorithms, has increased our ability to understand and extract information from scenic events. In most cases, Convolution neural networks (CNNs) are largely adopted to infer knowledge due to their surprising success in automation, surveillance, and many other application domains. However, the convolution operations’ overwhelming computation demand has somewhat limited their use in remote sensing edge devices. In these platforms, real-time processing remains a challenging task due to the tight constraints on resources and power. Here, the transfer and processing of non-relevant image pixels act as a bottleneck on the entire system. It is possible to overcome this bottleneck by exploiting the high bandwidth available at the sensor interface by designing a CNN inference architecture near the sensor. This paper presents an attention-based pixel processing architecture to facilitate the CNN inference near the image sensor. We propose an efficient computation method to reduce the dynamic power by decreasing the overall computation of the convolution operations. The proposed method reduces redundancies by using a hierarchical optimization approach. The approach minimizes power consumption for convolution operations by exploiting the Spatio-temporal redundancies found in the incoming feature maps and performs computations only on selected regions based on their relevance score. The proposed design addresses problems related to the mapping of computations onto an array of processing elements (PEs) and introduces a suitable network structure for communication. The PEs are highly optimized to provide low latency and power for CNN applications. While designing the model, we exploit the concepts of biological vision systems to reduce computation and energy. We prototype the model in a Virtex UltraScale+ FPGA and implement it in Application Specific Integrated Circuit (ASIC) using the TSMC 90nm technology library. The results suggest that the proposed architecture significantly reduces dynamic power consumption and achieves high-speed up surpassing existing embedded processors’ computational capabilities.

scholarly journals Prediction of Head Movement in 360-Degree Videos Using Attention Model

Sensors ◽  
2021 ◽  
Vol 21 (11) ◽  
pp. 3678
Author(s):  
Dongwon Lee ◽  
Minji Choi ◽  
Joohyun Lee
Keyword(s):  
Machine Learning ◽  
Short Term Memory ◽  
Moving Average ◽  
The Other ◽  
Learning Models ◽  
Short Term ◽  
Term Memory ◽  
Attention Model ◽  
Long Short Term Memory ◽  
Machine Learning Models

In this paper, we propose a prediction algorithm, the combination of Long Short-Term Memory (LSTM) and attention model, based on machine learning models to predict the vision coordinates when watching 360-degree videos in a Virtual Reality (VR) or Augmented Reality (AR) system. Predicting the vision coordinates while video streaming is important when the network condition is degraded. However, the traditional prediction models such as Moving Average (MA) and Autoregression Moving Average (ARMA) are linear so they cannot consider the nonlinear relationship. Therefore, machine learning models based on deep learning are recently used for nonlinear predictions. We use the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) neural network methods, originated in Recurrent Neural Networks (RNN), and predict the head position in the 360-degree videos. Therefore, we adopt the attention model to LSTM to make more accurate results. We also compare the performance of the proposed model with the other machine learning models such as Multi-Layer Perceptron (MLP) and RNN using the root mean squared error (RMSE) of predicted and real coordinates. We demonstrate that our model can predict the vision coordinates more accurately than the other models in various videos.

MULTIFRACTAL FLUCTUATIONS OF JIUZHAIGOU TOURISTS BEFORE AND AFTER WENCHUAN EARTHQUAKE

Fractals ◽  
2013 ◽  
Vol 21 (01) ◽  
pp. 1350001 ◽  
Author(s):  
KAI SHI ◽  
WEN-YONG LI ◽  
CHUN-QIONG LIU ◽  
ZHENG-WEN HUANG
Keyword(s):  
Wenchuan Earthquake ◽  
Moving Average ◽  
Long Term Memory ◽  
Short Term ◽  
Dynamical Mechanism ◽  
Before And After ◽  
Previous State ◽  
Multifractal Behavior ◽  
Multifractal Nature

In this work, multifractal methods have been successfully used to characterize the temporal fluctuations of daily Jiuzhai Valley domestic and foreign tourists before and after Wenchuan earthquake in China. We used multifractal detrending moving average method (MF-DMA). It showed that Jiuzhai Valley tourism markets are characterized by long-term memory and multifractal nature in. Moreover, the major sources of multifractality are studied. Based on the concept of sliding window, the time evolutions of the multifractal behavior of domestic and foreign tourists were analyzed and the influence of Wenchuan earthquake on Jiuzhai Valley tourism system dynamics were evaluated quantitatively. The study indicates that the inherent dynamical mechanism of Jiuzhai Valley tourism system has not been fundamentally changed from long views, although Jiuzhai Valley tourism system was seriously affected by the Wenchuan earthquake. Jiuzhai Valley tourism system has the ability to restore to its previous state in the short term.

Sign in / Sign up

Export Citation Format

Share Document