Parallel and distributed training of neural networks via successive convex approximation

2016 ◽  
Author(s):  
Paolo Di Lorenzo ◽  
Simone Scardapane
Keyword(s):  
Neural Networks ◽  
Convex Approximation ◽  
Distributed Training ◽  
Successive Convex Approximation

scholarly journals Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning

2019 ◽  
Vol 33 ◽  
pp. 5693-5700 ◽  
Author(s):  
Hao Yu ◽  
Sen Yang ◽  
Shenghuo Zhu
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Model Averaging ◽  
Communication Overhead ◽  
Single Server ◽  
Training Time ◽  
Distributed Training ◽  
Speed Up ◽  
Experimental Works ◽  
Single Worker

In distributed training of deep neural networks, parallel minibatch SGD is widely used to speed up the training process by using multiple workers. It uses multiple workers to sample local stochastic gradients in parallel, aggregates all gradients in a single server to obtain the average, and updates each worker’s local model using a SGD update with the averaged gradient. Ideally, parallel mini-batch SGD can achieve a linear speed-up of the training time (with respect to the number of workers) compared with SGD over a single worker. However, such linear scalability in practice is significantly limited by the growing demand for gradient communication as more workers are involved. Model averaging, which periodically averages individual models trained over parallel workers, is another common practice used for distributed training of deep neural networks since (Zinkevich et al. 2010) (McDonald, Hall, and Mann 2010). Compared with parallel mini-batch SGD, the communication overhead of model averaging is significantly reduced. Impressively, tremendous experimental works have verified that model averaging can still achieve a good speed-up of the training time as long as the averaging interval is carefully controlled. However, it remains a mystery in theory why such a simple heuristic works so well. This paper provides a thorough and rigorous theoretical study on why model averaging can work as well as parallel mini-batch SGD with significantly less communication overhead.

scholarly journals Power Gain Pattern Synthesis via Successive Convex Approximation Technique

IEEE Access ◽  
2020 ◽  
Vol 8 ◽  
pp. 181807-181814
Author(s):  
Feng Yang ◽  
Duyu Yi ◽  
Lingna Hu ◽  
Guangda Zang ◽  
Lianghui Ding
Keyword(s):  
Approximation Technique ◽  
Power Gain ◽  
Convex Approximation ◽  
Pattern Synthesis ◽  
Successive Convex Approximation
Sign in / Sign up

Export Citation Format

Share Document