Distributed mirror descent method for saddle point problems over directed graphs

Complexity ◽  
2016 ◽  
Vol 21 (S2) ◽  
pp. 178-190 ◽  
Author(s):  
Jueyou Li ◽  
Guo Chen ◽  
Zhaoyang Dong ◽  
Zhiyou Wu ◽  
Minghai Yao
Keyword(s):  
Saddle Point ◽  
Directed Graphs ◽  
Descent Method ◽  
Saddle Point Problems ◽  
Mirror Descent Method ◽  
Mirror Descent

Distributed mirror descent method for multi-agent optimization with delay

2016 ◽  
Vol 177 ◽  
pp. 643-650 ◽  
Author(s):  
Jueyou Li ◽  
Guo Chen ◽  
Zhaoyang Dong ◽  
Zhiyou Wu
Keyword(s):  
Descent Method ◽  
Mirror Descent Method ◽  
Mirror Descent ◽  
Multi Agent

Primal–Dual Mirror Descent Method for Constraint Stochastic Optimization Problems

2018 ◽  
Vol 58 (11) ◽  
pp. 1728-1736 ◽  
Author(s):  
A. S. Bayandina ◽  
A. V. Gasnikov ◽  
E. V. Gasnikova ◽  
S. V. Matsievskii
Keyword(s):  
Stochastic Optimization ◽  
Optimization Problems ◽  
Descent Method ◽  
Mirror Descent Method ◽  
Mirror Descent ◽  
Primal Dual

Stochastic mirror descent method for distributed multi-agent optimization

2016 ◽  
Vol 12 (6) ◽  
pp. 1179-1197 ◽  
Author(s):  
Jueyou Li ◽  
Guoquan Li ◽  
Zhiyou Wu ◽  
Changzhi Wu
Keyword(s):  
Descent Method ◽  
Mirror Descent Method ◽  
Mirror Descent ◽  
Multi Agent

scholarly journals Algorithms of Robust Stochastic Optimization Based on Mirror Descent Method

2019 ◽  
Vol 80 (9) ◽  
pp. 1607-1627 ◽  
Author(s):  
A. V. Nazin ◽  
A. S. Nemirovsky ◽  
A. B. Tsybakov ◽  
A. B. Juditsky
Keyword(s):  
Stochastic Optimization ◽  
Descent Method ◽  
Mirror Descent Method ◽  
Mirror Descent

scholarly journals Efficient Adaptive Online Learning via Frequent Directions

2018 ◽  
Author(s):  
Yuanyu Wan ◽  
Nan Wei ◽  
Lijun Zhang
Keyword(s):  
Online Learning ◽  
Large Scale ◽  
Descent Method ◽  
Low Rank ◽  
Subgradient Methods ◽  
Outer Product ◽  
Mirror Descent Method ◽  
Mirror Descent ◽  
Primal Dual ◽  
Frequent Directions

By employing time-varying proximal functions, adaptive subgradient methods (ADAGRAD) have improved the regret bound and been widely used in online learning and optimization. However, ADAGRAD with full matrix proximal functions (ADA-FULL) cannot deal with large-scale problems due to the impractical time and space complexities, though it has better performance when gradients are correlated. In this paper, we propose ADA-FD, an efficient variant of ADA-FULL based on a deterministic matrix sketching technique called frequent directions. Following ADA-FULL, we incorporate our ADA-FD into both primal-dual subgradient method and composite mirror descent method to develop two efficient methods. By maintaining and manipulating low-rank matrices, at each iteration, the space complexity is reduced from $O(d^2)$ to $O(\tau d)$ and the time complexity is reduced from $O(d^3)$ to $O(\tau^2d)$, where $d$ is the dimensionality of the data and $\tau \ll d$ is the sketching size. Theoretical analysis reveals that the regret of our methods is close to that of ADA-FULL as long as the outer product matrix of gradients is approximately low-rank. Experimental results show that our ADA-FD is comparable to ADA-FULL and outperforms other state-of-the-art algorithms in online convex optimization as well as in training convolutional neural networks (CNN).

Sign in / Sign up

Export Citation Format

Share Document