Recurrent Neural Networks for Edge Intelligence

2021 ◽  
Vol 54 (4) ◽  
pp. 1-38
Author(s):  
Varsha S. Lalapura ◽  
J. Amudha ◽  
Hariramn Selvamuruga Satheesh
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Model Performance ◽  
Network Architectures ◽  
Training Procedure ◽  
Resource Constrained ◽  
Compression Technique ◽  
Creative Art ◽  
Comprehensive Survey ◽  
Intelligent Models

Recurrent Neural Networks are ubiquitous and pervasive in many artificial intelligence applications such as speech recognition, predictive healthcare, creative art, and so on. Although they provide accurate superior solutions, they pose a massive challenge “training havoc.” Current expansion of IoT demands intelligent models to be deployed at the edge. This is precisely to handle increasing model sizes and complex network architectures. Design efforts to meet these for greater performance have had inverse effects on portability on edge devices with real-time constraints of memory, latency, and energy. This article provides a detailed insight into various compression techniques widely disseminated in the deep learning regime. They have become key in mapping powerful RNNs onto resource-constrained devices. While compression of RNNs is the main focus of the survey, it also highlights challenges encountered while training. The training procedure directly influences model performance and compression alongside. Recent advancements to overcome the training challenges with their strengths and drawbacks are discussed. In short, the survey covers the three-step process, namely, architecture selection, efficient training process, and suitable compression technique applicable to a resource-constrained environment. It is thus one of the comprehensive survey guides a developer can adapt for a time-series problem context and an RNN solution for the edge.

RAP: A Software Framework of Developing Convolutional Neural Networks for Resource-constrained Devices Using Environmental Monitoring as a Case Study

2021 ◽  
Vol 5 (4) ◽  
pp. 1-28
Author(s):  
Chia-Heng Tu ◽  
Qihui Sun ◽  
Hsiao-Hsuan Chang
Keyword(s):  
Neural Networks ◽  
Environmental Monitoring ◽  
Convolutional Neural Networks ◽  
Model Building ◽  
Software Framework ◽  
Training Procedure ◽  
Monitoring Methods ◽  
Resource Constrained ◽  
Resource Constrained Devices ◽  
Constrained Devices

Monitoring environmental conditions is an important application of cyber-physical systems. Typically, the monitoring is to perceive surrounding environments with battery-powered, tiny devices deployed in the field. While deep learning-based methods, especially the convolutional neural networks (CNNs), are promising approaches to enriching the functionalities offered by the tiny devices, they demand more computation and memory resources, which makes these methods difficult to be adopted on such devices. In this article, we develop a software framework, RAP , that permits the construction of the CNN designs by aggregating the existing, lightweight CNN layers, which are able to fit in the limited memory (e.g., several KBs of SRAM) on the resource-constrained devices satisfying application-specific timing constrains. RAP leverages the Python-based neural network framework Chainer to build the CNNs by mounting the C/C++ implementations of the lightweight layers, trains the built CNN models as the ordinary model-training procedure in Chainer, and generates the C version codes of the trained models. The generated programs are compiled into target machine executables for the on-device inferences. With the vigorous development of lightweight CNNs, such as binarized neural networks with binary weights and activations, RAP facilitates the model building process for the resource-constrained devices by allowing them to alter, debug, and evaluate the CNN designs over the C/C++ implementation of the lightweight CNN layers. We have prototyped the RAP framework and built two environmental monitoring applications for protecting endangered species using image- and acoustic-based monitoring methods. Our results show that the built model consumes less than 0.5 KB of SRAM for buffering the runtime data required by the model inference while achieving up to 93% of accuracy for the acoustic monitoring with less than one second of inference time on the TI 16-bit microcontroller platform.

scholarly journals CommNets: Communicating Neural Network Architectures for Resource Constrained Systems

2019 ◽  
Vol 33 ◽  
pp. 9909-9910
Author(s):  
Prince M Abudu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Network Architecture ◽  
Resource Constraints ◽  
Constrained Systems ◽  
Detection Accuracy ◽  
Network Architectures ◽  
Operating Environments ◽  
Hidden States

Applications that require heterogeneous sensor deployments continue to face practical challenges owing to resource constraints within their operating environments (i.e. energy efficiency, computational power and reliability). This has motivated the need for effective ways of selecting a sensing strategy that maximizes detection accuracy for events of interest using available resources and data-driven approaches. Inspired by those limitations, we ask a fundamental question: whether state-of-the-art Recurrent Neural Networks can observe different series of data and communicate their hidden states to collectively solve an objective in a distributed fashion. We realize our answer by conducting a series of systematic analyses of a Communicating Recurrent Neural Network architecture on varying time-steps, objective functions and number of nodes. The experimental setup we employ models tasks synonymous with those in Wireless Sensor Networks. Our contributions show that Recurrent Neural Networks can communicate through their hidden states and we achieve promising results.

scholarly journals Deep Sparse Learning for Automatic Modulation Classification Using Recurrent Neural Networks

Sensors ◽  
2021 ◽  
Vol 21 (19) ◽  
pp. 6410
Author(s):  
Ke Zang ◽  
Wenqi Wu ◽  
Wei Luo
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Recurrent Neural Networks ◽  
Deep Neural Networks ◽  
Learning Algorithm ◽  
Model Performance ◽  
Sparse Learning ◽  
Modulation Classification ◽  
Learning Models ◽  
Automatic Modulation Classification

Deep learning models, especially recurrent neural networks (RNNs), have been successfully applied to automatic modulation classification (AMC) problems recently. However, deep neural networks are usually overparameterized, i.e., most of the connections between neurons are redundant. The large model size hinders the deployment of deep neural networks in applications such as Internet-of-Things (IoT) networks. Therefore, reducing parameters without compromising the network performance via sparse learning is often desirable since it can alleviates the computational and storage burdens of deep learning models. In this paper, we propose a sparse learning algorithm that can directly train a sparsely connected neural network based on the statistics of weight magnitude and gradient momentum. We first used the MNIST and CIFAR10 datasets to demonstrate the effectiveness of this method. Subsequently, we applied it to RNNs with different pruning strategies on recurrent and non-recurrent connections for AMC problems. Experimental results demonstrated that the proposed method can effectively reduce the parameters of the neural networks while maintaining model performance. Moreover, we show that appropriate sparsity can further improve network generalization ability.

scholarly journals Predicting Protein Binding Affinity With Word Embeddings and Recurrent Neural Networks

2017 ◽  
Author(s):  
Carlo Mazzaferro
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Antigenic Determinants ◽  
Underlying Structure ◽  
Specific Response ◽  
Network Architectures ◽  
Learning Approaches ◽  
Word Embeddings ◽  
Viral Pathogens ◽  
Histocompatibility Complex

AbstractAt the core of our immunological system lies a group of proteins named Major Histocompatibility Complex (MHC), to which epitopes (also proteins sometimes named antigenic determinants), bind to eliciting a response. These responses are extremely varied and of widely different nature. For instance, Killer and Helper T cells are responsible for, respectively, counteracting viral pathogens and tumorous cells. Many other types exist, but their underlying structure can be very similar due to the fact that they all are proteins and bind to the MHC receptor in a similar fashion. With this framework in mind, being able to predict with precision the structure of a protein that will elicit a specific response in the human body represents a novel computational approach to drug discovery. Although many machine learning approaches have been used, no attempt to solve this problem using Recurrent Neural Networks (RNNs) exist. We extend the current efforts in the field by applying a variety of network architectures based on RNNs and word embeddings (WE). The code is freely available and under current development at https://github.com/carlomazzaferro/mhcPreds

Hydrological modelling using artificial neural networks

2001 ◽  
Vol 25 (1) ◽  
pp. 80-108 ◽  
Author(s):  
C. W. Dawson ◽  
R. L. Wilby
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Model Performance ◽  
Model Complexity ◽  
Network Architectures ◽  
Training Algorithms ◽  
Rainfall Runoff ◽  
Ann Model ◽  
Validation Data ◽  
Artificial Neural

This review considers the application of artificial neural networks (ANNs) to rainfall-runoff modelling and flood forecasting. This is an emerging field of research, characterized by a wide variety of techniques, a diversity of geographical contexts, a general absence of intermodel comparisons, and inconsistent reporting of model skill. This article begins by outlining the basic principles of ANN modelling, common network architectures and training algorithms. The discussion then addresses related themes of the division and preprocessing of data for model calibration/validation; data standardization techniques; and methods of evaluating ANN model performance. A literature survey underlines the need for clear guidance in current modelling practice, as well as the comparison of ANN methods with more conventional statistical models. Accordingly, a template is proposed in order to assist the construction of future ANN rainfall-runoff models. Finally, it is suggested that research might focus on the extraction of hydrological ‘rules’ from ANN weights, and on the development of standard performance measures that penalize unnecessary model complexity.

Echo-State Networks for Predicting ENSO Beyond One Year

2021 ◽  
Author(s):  
forough hassanibesheli ◽  
Niklas Boers ◽  
Jurgen Kurths
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Time Series ◽  
Recurrent Neural Networks ◽  
Southern Oscillation ◽  
Numerical Models ◽  
Weather Prediction ◽  
Supervised Machine Learning ◽  
Climate Indices ◽  
Training Procedure

<p>Most forecasting schemes in the geosciences, and in particular for predicting weather and<br>climate indices such as the El Niño Southern Oscillation (ENSO), rely on process-based<br>numerical models [1]. Although statistical modelling[2] and prediction approaches also have<br>a long history, more recently, different machine learning techniques have been used to predict<br>climatic time series. One of the supervised machine learning algorithm which is suited for<br>temporal and sequential data processing and prediction is given by recurrent neural networks<br>(RNNs)[3]. In this study we develop a RNN-based method that (1) can learn the dynamics<br>of a stochastic time series without requiring access to a huge amount of data for training, and<br>(2) has comparatively simple structure and efficient training procedure. Since this algorithm<br>is suitable for investigating complex nonlinear time series such as climate time series, we<br>apply it to different ENSO indices. We demonstrate that our model can capture key features<br>of the complex system dynamics underlying ENSO variability, and that it can accurately<br>forecast ENSO for longer lead times in comparison to other recent studies[4].</p><p> </p><p>Reference:</p><p>[1] P. Bauer, A. Thorpe, and G. Brunet, “The quiet revolution of numerical weather prediction,”<br>Nature, vol. 525, no. 7567, pp. 47–55, 2015.</p><p>[2] D. Kondrashov, S. Kravtsov, A. W. Robertson, and M. Ghil, “A hierarchy of data-based enso<br>models,” Journal of climate, vol. 18, no. 21, pp. 4425–4444, 2005.</p><p>[3] L. R. Medsker and L. Jain, “Recurrent neural networks,” Design and Applications, vol. 5, 2001.</p><p>[4] Y.-G. Ham, J.-H. Kim, and J.-J. Luo, “Deep learning for multi-year enso forecasts,” Nature,<br>vol. 573, no. 7775, pp. 568–572, 2019.</p>

scholarly journals Advances in Convolutional Neural Networks

2020 ◽  
Author(s):  
Wen Xu ◽  
Jing He ◽  
Yanfeng Shu ◽  
Hui Zheng
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Recurrent Neural Networks ◽  
Kernel Methods ◽  
Representation Learning ◽  
Multilayer Perceptrons ◽  
Learning Tasks ◽  
Comprehensive Survey ◽  
High Level ◽  
Advanced Applications

Deep Learning, also known as deep representation learning, has dramatically improved the performances on a variety of learning tasks and achieved tremendous successes in the past few years. Specifically, artificial neural networks are mainly studied, which mainly include Multilayer Perceptrons (MLPs), Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). Among these networks, CNNs got the most attention due to the kernel methods with the weight sharing mechanism, and achieved state-of-the-art in many domains, especially computer vision. In this research, we conduct a comprehensive survey related to the recent improvements in CNNs, and we demonstrate these advances from the low level to the high level, including the convolution operations, convolutional layers, architecture design, loss functions, and advanced applications.

scholarly journals Breathing-Based Authentication on Resource-Constrained IoT Devices using Recurrent Neural Networks

Computer ◽  
2018 ◽  
Vol 51 (5) ◽  
pp. 60-67 ◽  
Author(s):  
Jagmohan Chauhan ◽  
Suranga Seneviratne ◽  
Yining Hu ◽  
Archan Misra ◽  
Aruna Seneviratne ◽  
...  
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Resource Constrained ◽  
Iot Devices
Sign in / Sign up

Export Citation Format

Share Document