scholarly journals Pipelined Training with Stale Weights in Deep Convolutional Neural Networks

2021 ◽  
Vol 2021 ◽  
pp. 1-16
Author(s):  
Lifu Zhang ◽  
Tarek S. Abdelrahman
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Network Inference ◽  
Proof Of Concept ◽  
Statistical Efficiency ◽  
Deep Convolutional Neural Networks ◽  
Training Scheme ◽  
Memory Overhead ◽  
And Performance ◽  
The Impact

The growth in size and complexity of convolutional neural networks (CNNs) is forcing the partitioning of a network across multiple accelerators during training and pipelining of backpropagation computations over these accelerators. Pipelining results in the use of stale weights. Existing approaches to pipelined training avoid or limit the use of stale weights with techniques that either underutilize accelerators or increase training memory footprint. This paper contributes a pipelined backpropagation scheme that uses stale weights to maximize accelerator utilization and keep memory overhead modest. It explores the impact of stale weights on the statistical efficiency and performance using 4 CNNs (LeNet-5, AlexNet, VGG, and ResNet) and shows that when pipelining is introduced in early layers, training with stale weights converges and results in models with comparable inference accuracies to those resulting from nonpipelined training (a drop in accuracy of 0.4%, 4%, 0.83%, and 1.45% for the 4 networks, respectively). However, when pipelining is deeper in the network, inference accuracies drop significantly (up to 12% for VGG and 8.5% for ResNet-20). The paper also contributes a hybrid training scheme that combines pipelined with nonpipelined training to address this drop. The potential for performance improvement of the proposed scheme is demonstrated with a proof-of-concept pipelined backpropagation implementation in PyTorch on 2 GPUs using ResNet-56/110/224/362, achieving speedups of up to 1.8X over a 1-GPU baseline.

scholarly journals Comparing Object Recognition in Humans and Deep Convolutional Neural Networks—An Eye Tracking Study

2021 ◽  
Vol 15 ◽  
Author(s):  
Leonard Elia van Dyck ◽  
Roland Kwitt ◽  
Sebastian Jochen Denzler ◽  
Walter Roland Gruber
Keyword(s):  
Neural Networks ◽  
Object Recognition ◽  
Eye Tracking ◽  
Convolutional Neural Networks ◽  
Visual Pathway ◽  
Proof Of Concept ◽  
Deep Convolutional Neural Networks ◽  
Saliency Maps ◽  
Ventral Visual Pathway ◽  
Specific Factors

Deep convolutional neural networks (DCNNs) and the ventral visual pathway share vast architectural and functional similarities in visual challenges such as object recognition. Recent insights have demonstrated that both hierarchical cascades can be compared in terms of both exerted behavior and underlying activation. However, these approaches ignore key differences in spatial priorities of information processing. In this proof-of-concept study, we demonstrate a comparison of human observers (N = 45) and three feedforward DCNNs through eye tracking and saliency maps. The results reveal fundamentally different resolutions in both visualization methods that need to be considered for an insightful comparison. Moreover, we provide evidence that a DCNN with biologically plausible receptive field sizes called vNet reveals higher agreement with human viewing behavior as contrasted with a standard ResNet architecture. We find that image-specific factors such as category, animacy, arousal, and valence have a direct link to the agreement of spatial object recognition priorities in humans and DCNNs, while other measures such as difficulty and general image properties do not. With this approach, we try to open up new perspectives at the intersection of biological and computer vision research.

scholarly journals Optical Gas Sensing with Liquid Crystal Droplets and Convolutional Neural Networks

Sensors ◽  
2021 ◽  
Vol 21 (8) ◽  
pp. 2854
Author(s):  
José Frazão ◽  
Susana I. C. J. Palma ◽  
Henrique M. A. Costa ◽  
Cláudia Alves ◽  
Ana C. A. Roque ◽  
...  
Keyword(s):  
Neural Networks ◽  
Liquid Crystal ◽  
Convolutional Neural Networks ◽  
Low Cost ◽  
Droplet Diameter ◽  
Single Individual ◽  
Deep Convolutional Neural Networks ◽  
Optical Texture ◽  
Concentration Changes ◽  
The Impact

Liquid crystal (LC)-based materials are promising platforms to develop rapid, miniaturised and low-cost gas sensor devices. In hybrid gel films containing LC droplets, characteristic optical texture variations are observed due to orientational transitions of LC molecules in the presence of distinct volatile organic compounds (VOC). Here, we investigate the use of deep convolutional neural networks (CNN) as pattern recognition systems to analyse optical textures dynamics in LC droplets exposed to a set of different VOCs. LC droplets responses to VOCs were video recorded under polarised optical microscopy (POM). CNNs were then used to extract features from the responses and, in separate tasks, to recognise and quantify the vapours exposed to the films. The impact of droplet diameter on the results was also analysed. With our classification models, we show that a single individual droplet can recognise 11 VOCs with small structural and functional differences (F1-score above 93%). The optical texture variation pattern of a droplet also reflects VOC concentration changes, as suggested by applying a regression model to acetone at 0.9–4.0% (v/v) (mean absolute errors below 0.25% (v/v)). The CNN-based methodology is thus a promising approach for VOC sensing using responses from individual LC-droplets.

scholarly journals Golf swing classification with multiple deep convolutional neural networks

2018 ◽  
Vol 14 (10) ◽  
pp. 155014771880218 ◽  
Author(s):  
Libin Jiao ◽  
Rongfang Bie ◽  
Hao Wu ◽  
Yu Wei ◽  
Jixin Ma ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
Golf Swing ◽  
Support Vector ◽  
Classification Methods ◽  
Deep Convolutional Neural Networks ◽  
Machine Method ◽  
And Performance

The use of smart sports equipment and body sensory systems supervising daily sports training is gradually emerging in professional and amateur sports; however, the problem of processing large amounts of data from sensors used in sport and discovering constructive knowledge is a novel topic and the focus of our research. In this article, we investigate golf swing data classification methods based on varieties of representative convolutional neural networks (deep convolutional neural networks) which are fed with swing data from embedded multi-sensors, to group the multi-channel golf swing data labeled by hybrid categories from different golf players and swing shapes. In particular, four convolutional neural classifiers are customized: “GolfVanillaCNN” with the convolutional layers, “GolfVGG” with the stacked convolutional layers, “GolfInception” with the multi-scale convolutional layers, and “GolfResNet” with the residual learning. Testing on the real-world swing dataset sampled from the system integrating two strain gage sensors, three-axis accelerometer, and three-axis gyroscope, we explore the accuracy and performance of our convolutional neural network–based classifiers from two perspectives: classification implementations and sensor combinations. Besides, we further evaluate the performance of these four classifiers in terms of classification accuracy, precision–recall curves, and F1 scores. These common classification indicators illustrate that our convolutional neural network–based classifiers can basically group the golf swing predefined by the combination of shapes and golf players correctly and outperform support vector machine method representing traditional classification methods.

CNN-based Classification of Degraded Images

2020 ◽  
Vol 2020 (10) ◽  
pp. 28-1-28-7 ◽  
Author(s):  
Kazuki Endo ◽  
Masayuki Tanaka ◽  
Masatoshi Okutomi
Keyword(s):  
Neural Networks ◽  
Image Restoration ◽  
Image Classification ◽  
Convolutional Neural Networks ◽  
Deep Convolutional Neural Networks ◽  
Alternative Approach ◽  
Degraded Image ◽  
Degraded Images ◽  
Straightforward Approach

Classification of degraded images is very important in practice because images are usually degraded by compression, noise, blurring, etc. Nevertheless, most of the research in image classification only focuses on clean images without any degradation. Some papers have already proposed deep convolutional neural networks composed of an image restoration network and a classification network to classify degraded images. This paper proposes an alternative approach in which we use a degraded image and an additional degradation parameter for classification. The proposed classification network has two inputs which are the degraded image and the degradation parameter. The estimation network of degradation parameters is also incorporated if degradation parameters of degraded images are unknown. The experimental results showed that the proposed method outperforms a straightforward approach where the classification network is trained with degraded images only.

scholarly journals Deep convolutional neural networks for cardiovascular vulnerable plaque detection

2019 ◽  
Vol 277 ◽  
pp. 02024 ◽  
Author(s):  
Lincan Li ◽  
Tong Jia ◽  
Tianqi Meng ◽  
Yizhe Liu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Vulnerable Plaque ◽  
Recall Rate ◽  
Superior Performance ◽  
Learning Approaches ◽  
Deep Convolutional Neural Networks ◽  
Vulnerable Plaques ◽  
Plaque Detection

In this paper, an accurate two-stage deep learning method is proposed to detect vulnerable plaques in ultrasonic images of cardiovascular. Firstly, a Fully Convonutional Neural Network (FCN) named U-Net is used to segment the original Intravascular Optical Coherence Tomography (IVOCT) cardiovascular images. We experiment on different threshold values to find the best threshold for removing noise and background in the original images. Secondly, a modified Faster RCNN is adopted to do precise detection. The modified Faster R-CNN utilize six-scale anchors (122,162,322,642,1282,2562) instead of the conventional one scale or three scale approaches. First, we present three problems in cardiovascular vulnerable plaque diagnosis, then we demonstrate how our method solve these problems. The proposed method in this paper apply deep convolutional neural networks to the whole diagnostic procedure. Test results show the Recall rate, Precision rate, IoU (Intersection-over-Union) rate and Total score are 0.94, 0.885, 0.913 and 0.913 respectively, higher than the 1st team of CCCV2017 Cardiovascular OCT Vulnerable Plaque Detection Challenge. AP of the designed Faster RCNN is 83.4%, higher than conventional approaches which use one-scale or three-scale anchors. These results demonstrate the superior performance of our proposed method and the power of deep learning approaches in diagnose cardiovascular vulnerable plaques.

Sign in / Sign up

Export Citation Format

Share Document