Convergence analysis of the deep neural networks based globalized dual heuristic programming

Automatica ◽  
2020 ◽  
Vol 122 ◽  
pp. 109222
Author(s):  
Jong Woo Kim ◽  
Tae Hoon Oh ◽  
Sang Hwan Son ◽  
Dong Hwi Jeong ◽  
Jong Min Lee
Keyword(s):  
Neural Networks ◽  
Convergence Analysis ◽  
Deep Neural Networks ◽  
Heuristic Programming

scholarly journals Effects of depth, width, and initialization: A convergence analysis of layer-wise training for deep linear neural networks

2021 ◽  
pp. 1-47
Author(s):  
Yeonjong Shin
Keyword(s):  
Neural Networks ◽  
Convergence Analysis ◽  
Deep Neural Networks ◽  
Computational Cost ◽  
Back Propagation ◽  
Single Layer ◽  
Intermediate Layers ◽  
Machine Learning Applications ◽  
Gradient Based ◽  
Accelerate Convergence

Deep neural networks have been used in various machine learning applications and achieved tremendous empirical successes. However, training deep neural networks is a challenging task. Many alternatives have been proposed in place of end-to-end back-propagation. Layer-wise training is one of them, which trains a single layer at a time, rather than trains the whole layers simultaneously. In this paper, we study a layer-wise training using a block coordinate gradient descent (BCGD) for deep linear networks. We establish a general convergence analysis of BCGD and found the optimal learning rate, which results in the fastest decrease in the loss. We identify the effects of depth, width, and initialization. When the orthogonal-like initialization is employed, we show that the width of intermediate layers plays no role in gradient-based training beyond a certain threshold. Besides, we found that the use of deep networks could drastically accelerate convergence when it is compared to those of a depth 1 network, even when the computational cost is considered. Numerical examples are provided to justify our theoretical findings and demonstrate the performance of layer-wise training by BCGD.

Deep neural networks trained with heavier data augmentation learn features closer to representations in hIT

2018 ◽  
Author(s):  
Alex Hernández-García ◽  
Johannes Mehrer ◽  
Nikolaus Kriegeskorte ◽  
Peter König ◽  
Tim C. Kietzmann
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Data Augmentation

Representation of adversarial images in deep neural networks and the human brain

2018 ◽  
Author(s):  
Chi Zhang ◽  
Xiaohan Duan ◽  
Ruyuan Zhang ◽  
Li Tong
Keyword(s):  
Neural Networks ◽  
Human Brain ◽  
Deep Neural Networks
Sign in / Sign up

Export Citation Format

Share Document