Dynamic Regularization on Activation Sparsity for Neural Network Efficiency Improvement

2021 ◽  
Vol 17 (4) ◽  
pp. 1-16
Author(s):  
Qing Yang ◽  
Jiachen Mao ◽  
Zuoguan Wang ◽  
“Helen” Li Hai
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Low Cost ◽  
Data Communication ◽  
Network Efficiency ◽  
Computation Cost ◽  
Execution Speed ◽  
Static Feature ◽  
Weight Pruning ◽  
Sparsity Level

When deploying deep neural networks in embedded systems, it is crucial to decrease the model size and computational complexity for improving the execution speed and efficiency. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can also dramatically reduce the amount of data communication and the computation cost. Unlike weight parameters, the pattern of activations is directly related to input data and thereby changes dynamically. To regulate the dynamic activation sparsity (DAS), in this work, we propose a generic low-cost approach based on winners-take-all (WTA) dropout technique. The network enhanced by the proposed WTA dropout, namely DASNet , features structured activation sparsity with an improved sparsity level. Compared to the static feature map pruning methods, DASNets provide better computation cost reduction. The WTA dropout technique can be easily applied in deep neural networks without incurring additional training variables. More importantly, DASNet can be seamlessly integrated with other compression techniques, such as weight pruning and quantization, without compromising accuracy. Our experiments on various networks and datasets present significant runtime speedups with negligible accuracy losses.

scholarly journals SNR: S queezing N umerical R ange Defuses Bit Error Vulnerability Surface in Deep Neural Networks

2021 ◽  
Vol 20 (5s) ◽  
pp. 1-25
Author(s):  
Elbruz Ozen ◽  
Alex Orailoglu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Low Cost ◽  
Error Resilience ◽  
Error Rates ◽  
Training Methods ◽  
Performance Requirements ◽  
Proper Design ◽  
And Performance

As deep learning algorithms are widely adopted, an increasing number of them are positioned in embedded application domains with strict reliability constraints. The expenditure of significant resources to satisfy performance requirements in deep neural network accelerators has thinned out the margins for delivering safety in embedded deep learning applications, thus precluding the adoption of conventional fault tolerance methods. The potential of exploiting the inherent resilience characteristics of deep neural networks remains though unexplored, offering a promising low-cost path towards safety in embedded deep learning applications. This work demonstrates the possibility of such exploitation by juxtaposing the reduction of the vulnerability surface through the proper design of the quantization schemes with shaping the parameter distributions at each layer through the guidance offered by appropriate training methods, thus delivering deep neural networks of high resilience merely through algorithmic modifications. Unequaled error resilience characteristics can be thus injected into safety-critical deep learning applications to tolerate bit error rates of up to at absolutely zero hardware, energy, and performance costs while improving the error-free model accuracy even further.

Automated Log-Scale Quantization for Low-Cost Deep Neural Networks

2021 ◽  
Author(s):  
Sangyun Oh ◽  
Hyeonuk Sim ◽  
Sugil Lee ◽  
Jongeun Lee
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Low Cost

scholarly journals Intelligent control of quad-rotor aircrafts with a STM32 microcontroller using deep neural networks

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Xiaochun Guan ◽  
Sheng Lou ◽  
Han Li ◽  
Tinglong Tang
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Data Communication ◽  
Efficient Design ◽  
Embedded Devices ◽  
Content Type ◽  
Design Scheme ◽  
Communication Module ◽  
Aerial Vehicle ◽  
Optical Flow Sensor

Purpose Deployment of deep neural networks on embedded devices is becoming increasingly popular because it can reduce latency and energy consumption for data communication. This paper aims to give out a method for deployment the deep neural networks on a quad-rotor aircraft for further expanding its application scope. Design/methodology/approach In this paper, a design scheme is proposed to implement the flight mission of the quad-rotor aircraft based on multi-sensor fusion. It integrates attitude acquisition module, global positioning system position acquisition module, optical flow sensor, ultrasonic sensor and Bluetooth communication module, etc. A 32-bit microcontroller is adopted as the main controller for the quad-rotor aircraft. To make the quad-rotor aircraft be more intelligent, the study also proposes a method to deploy the pre-trained deep neural networks model on the microcontroller based on the software packages of the RT-Thread internet of things operating system. Findings This design provides a simple and efficient design scheme to further integrate artificial intelligence (AI) algorithm for the control system design of quad-rotor aircraft. Originality/value This method provides an application example and a design reference for the implementation of AI algorithms on unmanned aerial vehicle or terminal robots.

scholarly journals Deep Neural Networks for Low-Cost Eye Tracking

2020 ◽  
Vol 176 ◽  
pp. 685-694
Author(s):  
Ildar Rakhmatulin ◽  
Andrew T. Duchowski
Keyword(s):  
Neural Networks ◽  
Eye Tracking ◽  
Deep Neural Networks ◽  
Low Cost

scholarly journals Differential Evolution Based Layer-Wise Weight Pruning for Compressing Deep Neural Networks

Sensors ◽  
2021 ◽  
Vol 21 (3) ◽  
pp. 880
Author(s):  
Tao Wu ◽  
Xiaoyang Li ◽  
Deyun Zhou ◽  
Na Li ◽  
Jiao Shi
Keyword(s):  
Neural Networks ◽  
Differential Evolution ◽  
Deep Neural Networks ◽  
Experimental Studies ◽  
Fine Tuning ◽  
Sensor Data ◽  
The Past ◽  
Sensor Platforms ◽  
Resource Limited ◽  
Weight Pruning

Deep neural networks have evolved significantly in the past decades and are now able to achieve better progression of sensor data. Nonetheless, most of the deep models verify the ruling maxim in deep learning—bigger is better—so they have very complex structures. As the models become more complex, the computational complexity and resource consumption of these deep models are increasing significantly, making them difficult to perform on resource-limited platforms, such as sensor platforms. In this paper, we observe that different layers often have different pruning requirements, and propose a differential evolutionary layer-wise weight pruning method. Firstly, the pruning sensitivity of each layer is analyzed, and then the network is compressed by iterating the weight pruning process. Unlike some other methods that deal with pruning ratio by greedy ways or statistical analysis, we establish an optimization model to find the optimal pruning sensitivity set for each layer. Differential evolution is an effective method based on population optimization which can be used to address this task. Furthermore, we adopt a strategy to recovery some of the removed connections to increase the capacity of the pruned model during the fine-tuning phase. The effectiveness of our method has been demonstrated in experimental studies. Our method compresses the number of weight parameters in LeNet-300-100, LeNet-5, AlexNet and VGG16 by 24×, 14×, 29× and 12×, respectively.

scholarly journals User Driven FPGA-Based Design Automated Framework of Deep Neural Networks For Low-Power Low-Cost Edge Computing

IEEE Access ◽  
2021 ◽  
pp. 1-1
Author(s):  
TaRek Belabed ◽  
Maria Gracielly F. Coutinho ◽  
Marcelo A. C. Fernandes ◽  
Carlos Valderrama ◽  
Chokri Souani
Keyword(s):  
Neural Networks ◽  
Low Power ◽  
Deep Neural Networks ◽  
Low Cost ◽  
Edge Computing

scholarly journals Sparse Adversarial Perturbations for Videos

2019 ◽  
Vol 33 ◽  
pp. 8973-8980 ◽  
Author(s):  
Xingxing Wei ◽  
Jun Zhu ◽  
Sha Yuan ◽  
Hang Su
Keyword(s):  
Neural Networks ◽  
Action Recognition ◽  
Optimization Algorithm ◽  
Deep Neural Networks ◽  
Spatial Cues ◽  
Current Frame ◽  
Computation Cost ◽  
Temporal Cues ◽  
Static Images ◽  
Temporal Interactions

Although adversarial samples of deep neural networks (DNNs) have been intensively studied on static images, their extensions in videos are never explored. Compared with images, attacking a video needs to consider not only spatial cues but also temporal cues. Moreover, to improve the imperceptibility as well as reduce the computation cost, perturbations should be added on as few frames as possible, i.e., adversarial perturbations are temporally sparse. This further motivates the propagation of perturbations, which denotes that perturbations added on the current frame can transfer to the next frames via their temporal interactions. Thus, no (or few) extra perturbations are needed for these frames to misclassify them. To this end, we propose the first white-box video attack method, which utilizes an l2,1-norm based optimization algorithm to compute the sparse adversarial perturbations for videos. We choose the action recognition as the targeted task, and networks with a CNN+RNN architecture as threat models to verify our method. Thanks to the propagation, we can compute perturbations on a shortened version video, and then adapt them to the long version video to fool DNNs. Experimental results on the UCF101 dataset demonstrate that even only one frame in a video is perturbed, the fooling rate can still reach 59.7%.

scholarly journals StochasticNet in StochasticNet

2016 ◽  
Vol 2 (1) ◽  
Author(s):  
Mohammad Javad Shafiee ◽  
Paul Fieguth ◽  
Alexander Wong
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
High Performance ◽  
Deep Neural Network ◽  
Deep Neural Networks ◽  
Low Cost ◽  
Network Architectures ◽  
Modeling Accuracy ◽  
Random Graph Theory ◽  
Performance Computing

Deep neural networks have been shown to outperform conventionalstate-of-the-art approaches in several structured predictionapplications. While high-performance computing devices such asGPUs has made developing very powerful deep neural networkspossible, it is not feasible to run these networks on low-cost, lowpowercomputing devices such as embedded CPUs or even embeddedGPUs. As such, there has been a lot of recent interestto produce efficient deep neural network architectures that can berun on small computing devices. Motivated by this, the idea ofStochasticNets was introduced, where deep neural networks areformed by leveraging random graph theory. It has been shownthat StochasticNet can form new networks with 2X or 3X architecturalefficiency while maintaining modeling accuracy. Motivated bythese promising results, here we investigate the idea of Stochastic-Net in StochasticNet (SiS), where highly-efficient deep neural networkswith Network in Network (NiN) architectures are formed ina stochastic manner. Such networks have an intertwining structurecomposed of convolutional layers and micro neural networksto boost the modeling accuracy. The experimental results showthat SiS can form deep neural networks with NiN architectures thathave 4X greater architectural efficiency with only a 2% dropin accuracy for the CIFAR10 dataset. The results are even morepromising for the SVHN dataset, where SiS formed deep neuralnetworks with NiN architectures that have 11.5X greater architecturalefficiency with only a 1% decrease in modeling accuracy.

Sign in / Sign up

Export Citation Format

Share Document