scholarly journals A Hierarchical Deep Fusion Framework for Egocentric Activity Recognition Using a Wearable Hybrid Sensor System

Sensors ◽  
2019 ◽  
Vol 19 (3) ◽  
pp. 546 ◽  
Author(s):  
Haibin Yu ◽  
Guoxiong Pan ◽  
Mian Pan ◽  
Chong Li ◽  
Wenyan Jia ◽  
...  
Keyword(s):  
Activity Recognition ◽  
Short Term Memory ◽  
Specific Activity ◽  
Negative Influence ◽  
Sensor System ◽  
Sensor Data ◽  
Motion Sensor ◽  
Motion Sensors ◽  
Motion State ◽  
Fusion Framework

Recently, egocentric activity recognition has attracted considerable attention in the pattern recognition and artificial intelligence communities because of its wide applicability in medical care, smart homes, and security monitoring. In this study, we developed and implemented a deep-learning-based hierarchical fusion framework for the recognition of egocentric activities of daily living (ADLs) in a wearable hybrid sensor system comprising motion sensors and cameras. Long short-term memory (LSTM) and a convolutional neural network are used to perform egocentric ADL recognition based on motion sensor data and photo streaming in different layers, respectively. The motion sensor data are used solely for activity classification according to motion state, while the photo stream is used for further specific activity recognition in the motion state groups. Thus, both motion sensor data and photo stream work in their most suitable classification mode to significantly reduce the negative influence of sensor differences on the fusion results. Experimental results show that the proposed method not only is more accurate than the existing direct fusion method (by up to 6%) but also avoids the time-consuming computation of optical flow in the existing method, which makes the proposed algorithm less complex and more suitable for practical application.

scholarly journals Similarity Embedding Networks for Robust Human Activity Recognition

2021 ◽  
Vol 15 (6) ◽  
pp. 1-17
Author(s):  
Chenglin Li ◽  
Carrie Lu Tong ◽  
Di Niu ◽  
Bei Jiang ◽  
Xiao Zuo ◽  
...  
Keyword(s):  
Activity Recognition ◽  
Human Activity ◽  
Short Term Memory ◽  
Real Space ◽  
Human Activity Recognition ◽  
Sensor Data ◽  
Activity Data ◽  
Extensive Evaluation ◽  
Sensor Signals ◽  
Public Datasets

Deep learning models for human activity recognition (HAR) based on sensor data have been heavily studied recently. However, the generalization ability of deep models on complex real-world HAR data is limited by the availability of high-quality labeled activity data, which are hard to obtain. In this article, we design a similarity embedding neural network that maps input sensor signals onto real vectors through carefully designed convolutional and Long Short-Term Memory (LSTM) layers. The embedding network is trained with a pairwise similarity loss, encouraging the clustering of samples from the same class in the embedded real space, and can be effectively trained on a small dataset and even on a noisy dataset with mislabeled samples. Based on the learned embeddings, we further propose both nonparametric and parametric approaches for activity recognition. Extensive evaluation based on two public datasets has shown that the proposed similarity embedding network significantly outperforms state-of-the-art deep models on HAR classification tasks, is robust to mislabeled samples in the training set, and can also be used to effectively denoise a noisy dataset.

scholarly journals Wearable Sensor-Based Human Activity Recognition Using Hybrid Deep Learning Techniques

2020 ◽  
Vol 2020 ◽  
pp. 1-12 ◽  
Author(s):  
Huaijun Wang ◽  
Jing Zhao ◽  
Junhuai Li ◽  
Ling Tian ◽  
Pengjia Tu ◽  
...  
Keyword(s):  
Health Care ◽  
Deep Learning ◽  
Activity Recognition ◽  
Human Activity ◽  
Short Term Memory ◽  
Specific Activity ◽  
Recognition Rate ◽  
Human Activity Recognition ◽  
Wearable Sensor ◽  
Identification Rate

Human activity recognition (HAR) can be exploited to great benefits in many applications, including elder care, health care, rehabilitation, entertainment, and monitoring. Many existing techniques, such as deep learning, have been developed for specific activity recognition, but little for the recognition of the transitions between activities. This work proposes a deep learning based scheme that can recognize both specific activities and the transitions between two different activities of short duration and low frequency for health care applications. In this work, we first build a deep convolutional neural network (CNN) for extracting features from the data collected by sensors. Then, the long short-term memory (LTSM) network is used to capture long-term dependencies between two actions to further improve the HAR identification rate. By combing CNN and LSTM, a wearable sensor based model is proposed that can accurately recognize activities and their transitions. The experimental results show that the proposed approach can help improve the recognition rate up to 95.87% and the recognition rate for transitions higher than 80%, which are better than those of most existing similar models over the open HAPT dataset.

scholarly journals Mining Non-Zero-Rare Sequential Patterns On Activity Recognition

2019 ◽  
Vol 5 (1) ◽  
pp. 1-9
Author(s):  
Mohammad Iqbal ◽  
Chandrawati Putri Wulandari ◽  
Wawan Yunanto ◽  
Ghaluh Indah Permata Sari
Keyword(s):  
Activity Recognition ◽  
Human Activity ◽  
Human Activities ◽  
Activity Patterns ◽  
Human Activity Recognition ◽  
Experimental Result ◽  
Sequential Patterns ◽  
Motion Sensor ◽  
Motion Sensors ◽  
Human Activity Patterns

Discovering rare human activity patterns—from triggered motion sensors deliver peculiar information to notify people about hazard situations. This study aims to recognize rare human activities using mining non-zero-rare sequential patterns technique. In particular, this study mines the triggered motion sensor sequences to obtain non-zero-rare human activity patterns—the patterns which most occur in the motion sensor sequences and the occurrence numbers are less than the pre-defined occurrence threshold. This study proposes an algorithm to mine non-zero-rare pattern on human activity recognition called Mining Multi-class Non-Zero-Rare Sequential Patterns (MMRSP).  The experimental result showed that non-zero-rare human activity patterns succeed to capture the unusual activity. Furthermore, the MMRSP performed well according to the precision value of rare activities.

scholarly journals Human Activity Recognition for Indoor Localization Using Smartphone Inertial Sensors

Sensors ◽  
2021 ◽  
Vol 21 (18) ◽  
pp. 6316
Author(s):  
Dinis Moreira ◽  
Marília Barandas ◽  
Tiago Rocha ◽  
Pedro Alves ◽  
Ricardo Santos ◽  
...  
Keyword(s):  
Activity Recognition ◽  
Human Activity ◽  
Human Activities ◽  
Short Term Memory ◽  
Indoor Localization ◽  
Inertial Sensor ◽  
Human Activity Recognition ◽  
Location Based Services ◽  
Sensor Data ◽  
Indoor Environments

With the fast increase in the demand for location-based services and the proliferation of smartphones, the topic of indoor localization is attracting great interest. In indoor environments, users’ performed activities carry useful semantic information. These activities can then be used by indoor localization systems to confirm users’ current relative locations in a building. In this paper, we propose a deep-learning model based on a Convolutional Long Short-Term Memory (ConvLSTM) network to classify human activities within the indoor localization scenario using smartphone inertial sensor data. Results show that the proposed human activity recognition (HAR) model accurately identifies nine types of activities: not moving, walking, running, going up in an elevator, going down in an elevator, walking upstairs, walking downstairs, or going up and down a ramp. Moreover, predicted human activities were integrated within an existing indoor positioning system and evaluated in a multi-story building across several testing routes, with an average positioning error of 2.4 m. The results show that the inclusion of human activity information can reduce the overall localization error of the system and actively contribute to the better identification of floor transitions within a building. The conducted experiments demonstrated promising results and verified the effectiveness of using human activity-related information for indoor localization.

scholarly journals Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning

Sensors ◽  
2019 ◽  
Vol 19 (7) ◽  
pp. 1716 ◽  
Author(s):  
Seungeun Chung ◽  
Jiyoun Lim ◽  
Kyoung Ju Noh ◽  
Gague Kim ◽  
Hyuntae Jeong
Keyword(s):  
Deep Learning ◽  
Data Acquisition ◽  
Sensor Fusion ◽  
Activity Recognition ◽  
Human Activity ◽  
Short Term Memory ◽  
Sampling Rate ◽  
Human Activity Recognition ◽  
Sensor Data ◽  
Activity Data

In this paper, we perform a systematic study about the on-body sensor positioning and data acquisition details for Human Activity Recognition (HAR) systems. We build a testbed that consists of eight body-worn Inertial Measurement Units (IMU) sensors and an Android mobile device for activity data collection. We develop a Long Short-Term Memory (LSTM) network framework to support training of a deep learning model on human activity data, which is acquired in both real-world and controlled environments. From the experiment results, we identify that activity data with sampling rate as low as 10 Hz from four sensors at both sides of wrists, right ankle, and waist is sufficient in recognizing Activities of Daily Living (ADLs) including eating and driving activity. We adopt a two-level ensemble model to combine class-probabilities of multiple sensor modalities, and demonstrate that a classifier-level sensor fusion technique can improve the classification performance. By analyzing the accuracy of each sensor on different types of activity, we elaborate custom weights for multimodal sensor fusion that reflect the characteristic of individual activities.

scholarly journals Motion-To-BMI: Using Motion Sensors to Predict the Body Mass Index of Smartphone Users

Sensors ◽  
2020 ◽  
Vol 20 (4) ◽  
pp. 1134 ◽  
Author(s):  
Yumin Yao ◽  
Ling Song ◽  
Jin Ye
Keyword(s):  
Body Mass Index ◽  
Body Mass ◽  
Short Term Memory ◽  
Body Movement ◽  
The Body ◽  
Sensor Data ◽  
Motion Sensors ◽  
Sensitive Data ◽  
Public Datasets ◽  
Fully Connected

Obesity has become a widespread health problem worldwide. The body mass index (BMI) is a simple and reliable index based on weight and height that is commonly used to identify and classify adults as underweight, normal, overweight (pre-obesity), or obese. In this paper, we propose a hybrid deep neural network for predicting the BMI of smartphone users, based only on the characteristics of body movement captured by the smartphone’s built-in motion sensors without any other sensitive data. The proposed deep learning model consists of four major modules: a transformation module for data preprocessing, a convolution module for extracting spatial features, a long short-term memory (LSTM) module for exploring temporal dependency, and a fully connected module for regression. We define motion entropy (MEn), which is a measure of the regularity and complexity of the motion sensor, and propose a novel MEn-based filtering strategy to select parts of sensor data that met certain thresholds for training the model. We evaluate this model using two public datasets in comparison with baseline conventional feature-based methods using leave-one-subject-out (LOSO) cross-validation. Experimental results show that the proposed model with the MEn-based filtering strategy outperforms the baseline approaches significantly. The results also show that jogging may be a more suitable activity of daily living (ADL) for BMI prediction than walking and walking upstairs. We believe that the conclusions of this study will help to develop a long-term remote health monitoring system.

scholarly journals FilterNet: A Many-to-Many Deep Learning Architecture for Time Series Classification

Sensors ◽  
2020 ◽  
Vol 20 (9) ◽  
pp. 2498 ◽  
Author(s):  
Robert D. Chambers ◽  
Nathanael C. Yoder
Keyword(s):  
Time Series ◽  
Deep Learning ◽  
Activity Recognition ◽  
Short Term Memory ◽  
Sensor Data ◽  
Time Series Classification ◽  
Discrete Events ◽  
Segmentation Accuracy ◽  
Model Size

In this paper, we present and benchmark FilterNet, a flexible deep learning architecture for time series classification tasks, such as activity recognition via multichannel sensor data. It adapts popular convolutional neural network (CNN) and long short-term memory (LSTM) motifs which have excelled in activity recognition benchmarks, implementing them in a many-to-many architecture to markedly improve frame-by-frame accuracy, event segmentation accuracy, model size, and computational efficiency. We propose several model variants, evaluate them alongside other published models using the Opportunity benchmark dataset, demonstrate the effect of model ensembling and of altering key parameters, and quantify the quality of the models’ segmentation of discrete events. We also offer recommendations for use and suggest potential model extensions. FilterNet advances the state of the art in all measured accuracy and speed metrics when applied to the benchmarked dataset, and it can be extensively customized for other applications.

scholarly journals Sensor-Based Human Activity Recognition with Spatio-Temporal Deep Learning

Sensors ◽  
2021 ◽  
Vol 21 (6) ◽  
pp. 2141
Author(s):  
Ohoud Nafea ◽  
Wadood Abdul ◽  
Ghulam Muhammad ◽  
Mansour Alsulaiman
Keyword(s):  
Deep Learning ◽  
Activity Recognition ◽  
Human Activity ◽  
Short Term Memory ◽  
Human Activity Recognition ◽  
Sensor Data ◽  
Effective Selection ◽  
Spatio Temporal ◽  
High Level ◽  
Improved Accuracy

Human activity recognition (HAR) remains a challenging yet crucial problem to address in computer vision. HAR is primarily intended to be used with other technologies, such as the Internet of Things, to assist in healthcare and eldercare. With the development of deep learning, automatic high-level feature extraction has become a possibility and has been used to optimize HAR performance. Furthermore, deep-learning techniques have been applied in various fields for sensor-based HAR. This study introduces a new methodology using convolution neural networks (CNN) with varying kernel dimensions along with bi-directional long short-term memory (BiLSTM) to capture features at various resolutions. The novelty of this research lies in the effective selection of the optimal video representation and in the effective extraction of spatial and temporal features from sensor data using traditional CNN and BiLSTM. Wireless sensor data mining (WISDM) and UCI datasets are used for this proposed methodology in which data are collected through diverse methods, including accelerometers, sensors, and gyroscopes. The results indicate that the proposed scheme is efficient in improving HAR. It was thus found that unlike other available methods, the proposed method improved accuracy, attaining a higher score in the WISDM dataset compared to the UCI dataset (98.53% vs. 97.05%).

scholarly journals Improving the Authentication with Built-In Camera Protocol Using Built-In Motion Sensors: A Deep Learning Solution

Mathematics ◽  
2021 ◽  
Vol 9 (15) ◽  
pp. 1786
Author(s):  
Cezara Benegui ◽  
Radu Tudor Ionescu
Keyword(s):  
Deep Learning ◽  
Sensor Data ◽  
Support Vector ◽  
Motion Sensor ◽  
Qr Code ◽  
Motion Sensors ◽  
Probe Signal ◽  
Qr Codes ◽  
Social Media Platforms ◽  
Meta Information

In this paper, we propose an enhanced version of the Authentication with Built-in Camera (ABC) protocol by employing a deep learning solution based on built-in motion sensors. The standard ABC protocol identifies mobile devices based on the photo-response non-uniformity (PRNU) of the camera sensor, while also considering QR-code-based meta-information. During registration, users are required to capture photos using their smartphone camera. The photos are sent to a server that computes the camera fingerprint, storing it as an authentication trait. During authentication, the user is required to take two photos that contain two QR codes presented on a screen. The presented QR code images also contain a unique probe signal, similar to a camera fingerprint, generated by the protocol. During verification, the server computes the fingerprint of the received photos and authenticates the user if (i) the probe signal is present, (ii) the metadata embedded in the QR codes is correct and (iii) the camera fingerprint is identified correctly. However, the protocol is vulnerable to forgery attacks when the attacker can compute the camera fingerprint from external photos, as shown in our preliminary work. Hence, attackers can easily remove their PRNU from the authentication photos without completely altering the probe signal, resulting in attacks that bypass the defense systems of the ABC protocol. In this context, we propose an enhancement to the ABC protocol, using motion sensor data as an additional and passive authentication layer. Smartphones can be identified through their motion sensor data, which, unlike photos, is never posted by users on social media platforms, thus being more secure than using photographs alone. To this end, we transform motion signals into embedding vectors produced by deep neural networks, applying Support Vector Machines for the smartphone identification task. Our change to the ABC protocol results in a multi-modal protocol that lowers the false acceptance rate for the attack proposed in our previous work to a percentage as low as 0.07%. In this paper, we present the attack that makes ABC vulnerable, as well as our multi-modal ABC protocol along with relevant experiments and results.

scholarly journals Joint Learning of Temporal Models to Handle Imbalanced Data for Human Activity Recognition

2020 ◽  
Vol 10 (15) ◽  
pp. 5293 ◽  
Author(s):  
Rebeen Ali Hamad ◽  
Longzhi Yang ◽  
Wai Lok Woo ◽  
Bo Wei
Keyword(s):  
Deep Learning ◽  
Activity Recognition ◽  
Human Activity ◽  
Smart Home ◽  
Short Term Memory ◽  
Recognition System ◽  
Human Activity Recognition ◽  
Sensor Data ◽  
Temporal Models ◽  
Wide Range

Human activity recognition has become essential to a wide range of applications, such as smart home monitoring, health-care, surveillance. However, it is challenging to deliver a sufficiently robust human activity recognition system from raw sensor data with noise in a smart environment setting. Moreover, imbalanced human activity datasets with less frequent activities create extra challenges for accurate activity recognition. Deep learning algorithms have achieved promising results on balanced datasets, but their performance on imbalanced datasets without explicit algorithm design cannot be promised. Therefore, we aim to realise an activity recognition system using multi-modal sensors to address the issue of class imbalance in deep learning and improve recognition accuracy. This paper proposes a joint diverse temporal learning framework using Long Short Term Memory and one-dimensional Convolutional Neural Network models to improve human activity recognition, especially for less represented activities. We extensively evaluate the proposed method for Activities of Daily Living recognition using binary sensors dataset. A comparative study on five smart home datasets demonstrate that our proposed approach outperforms the existing individual temporal models and their hybridization. Furthermore, this is particularly the case for minority classes in addition to reasonable improvement on the majority classes of human activities.

Sign in / Sign up

Export Citation Format

Share Document