On the Unified Design of Accelerated Gradient Descent

2019 ◽  
Author(s):  
Yuquan Chen ◽  
Yiheng Wei ◽  
Yong Wang ◽  
YangQuan Chen
Keyword(s):  
Transfer Function ◽  
Gradient Descent ◽  
Design Procedure ◽  
Inverse Laplace Transform ◽  
Step Size ◽  
System Perspective ◽  
Infinite Dimensional ◽  
Numerical Inverse Laplace Transform ◽  
Dimensional Property ◽  
Accelerated Gradient

Abstract Nowadays, different kinds of problems such as modeling, optimal control, and machine learning can be formulated as an optimization problem. Gradient descent is the most popular method to solve such problem and many accelerated gradient descents have been designed to improve the performance. In this paper, we will analyze the basic gradient descent, momentum gradient descent, and Nesterov accelerated gradient descent from the system perspective and it is found that all of them can be formulated as a feedback control problem for tracking an extreme point. On this basis, a unified gradient descent design procedure is given, where a high order transfer function is considered. Furthermore, as an extension, both a fractional integrator and a general fractional transfer function are considered, which resulting in the fractional gradient descent. Due to the infinite-dimensional property of fractional order systems, numerical inverse Laplace transform and Matlab command stmcb() are used to realize a finite-order implementation for the fractional gradient descent. Besides the simplified design procedure, it is found that the convergence rate of fractional gradient descent is more robust to the step size by simulating results.

scholarly journals A Transformation of Accelerated Double Step Size Method for Unconstrained Optimization

2015 ◽  
Vol 2015 ◽  
pp. 1-8 ◽  
Author(s):  
Predrag S. Stanimirović ◽  
Gradimir V. Milovanović ◽  
Milena J. Petrović ◽  
Nataša Z. Kontrec
Keyword(s):  
Gradient Descent ◽  
Linear Convergence ◽  
Step Length ◽  
Descent Method ◽  
Single Step ◽  
Gradient Descent Method ◽  
Step Size ◽  
Double Step ◽  
Substantial Progress ◽  
Accelerated Gradient

A reduction of the originally double step size iteration into the single step length scheme is derived under the proposed condition that relates two step lengths in the accelerated double step size gradient descent scheme. The proposed transformation is numerically tested. Obtained results confirm the substantial progress in comparison with the single step size accelerated gradient descent method defined in a classical way regarding all analyzed characteristics: number of iterations, CPU time, and number of function evaluations. Linear convergence of derived method has been proved.

scholarly journals Meta-Descent for Online, Continual Prediction

2019 ◽  
Vol 33 ◽  
pp. 3943-3950
Author(s):  
Andrew Jacobsen ◽  
Matthew Schlegel ◽  
Cameron Linke ◽  
Thomas Degris ◽  
Adam White ◽  
...  
Keyword(s):  
Gradient Descent ◽  
Large Scale ◽  
Time Series Prediction ◽  
Real Data ◽  
Second Order ◽  
Stochastic Gradient Descent ◽  
Step Size ◽  
Vector Approximation ◽  
Prediction Problems ◽  
Stationary Problems

This paper investigates different vector step-size adaptation approaches for non-stationary online, continual prediction problems. Vanilla stochastic gradient descent can be considerably improved by scaling the update with a vector of appropriately chosen step-sizes. Many methods, including AdaGrad, RMSProp, and AMSGrad, keep statistics about the learning process to approximate a second order update—a vector approximation of the inverse Hessian. Another family of approaches use meta-gradient descent to adapt the stepsize parameters to minimize prediction error. These metadescent strategies are promising for non-stationary problems, but have not been as extensively explored as quasi-second order methods. We first derive a general, incremental metadescent algorithm, called AdaGain, designed to be applicable to a much broader range of algorithms, including those with semi-gradient updates or even those with accelerations, such as RMSProp. We provide an empirical comparison of methods from both families. We conclude that methods from both families can perform well, but in non-stationary prediction problems the meta-descent methods exhibit advantages. Our method is particularly robust across several prediction problems, and is competitive with the state-of-the-art method on a large-scale, time-series prediction problem on real data from a mobile robot.

Numerical Inversion of Laplace Transform for Time Resolved Thermal Characterization Experiment

2011 ◽  
Vol 133 (4) ◽  
Author(s):  
J. Toutain ◽  
J.-L. Battaglia ◽  
C. Pradere ◽  
J. Pailhes ◽  
A. Kusiak ◽  
...  
Keyword(s):  
Fourier Series ◽  
Laplace Transform ◽  
Periodic Function ◽  
Thermal Characterization ◽  
Inverse Laplace Transform ◽  
Numerical Inversion ◽  
Time Resolved ◽  
Reliable Technique ◽  
Numerical Inverse Laplace Transform ◽  
Thermal Disturbance

The aim of this technical brief is to test numerical inverse Laplace transform methods with application in the framework of the thermal characterization experiment. The objective is to find the most reliable technique in the case of a time resolved experiment based on a thermal disturbance in the form of a periodic function or a distribution. The reliability of methods based on the Fourier series methods is demonstrated.

On the approximate inverse Laplace transform of the transfer function with a single fractional order

2020 ◽  
pp. 014233122097766
Author(s):  
Ali Yüce ◽  
Nusret Tan
Keyword(s):  
Transfer Function ◽  
Laplace Transform ◽  
Fractional Order ◽  
Analytical Solutions ◽  
Transfer Functions ◽  
Inverse Laplace Transform ◽  
Fractional Order Systems ◽  
Approximate Inverse ◽  
Classical Mathematics ◽  
Curve Fitting Method

The history of fractional calculus dates back to 1600s and it is almost as old as classical mathematics. Although many studies have been published on fractional-order control systems in recent years, there is still a lack of analytical solutions. The focus of this study is to obtain analytical solutions for fractional order transfer functions with a single fractional element and unity coefficient. Approximate inverse Laplace transformation, that is, time response of the basic transfer function, is obtained analytically for the fractional order transfer functions with single-fractional-element by curve fitting method. Obtained analytical equations are tabulated for some fractional orders of [Formula: see text]. Moreover, a single function depending on fractional order alpha has been introduced for the first time using a table for [Formula: see text]. By using this table, approximate inverse Laplace transform function is obtained in terms of any fractional order of [Formula: see text] for [Formula: see text]. Obtained analytic equations offer accurate results in computing inverse Laplace transforms. The accuracy of the method is supported by numerical examples in this study. Also, the study sets the basis for the higher fractional-order systems that can be decomposed into a single (simpler) fractional order systems.

scholarly journals Periodic step-size adaptation in second-order gradient descent for single-pass on-line structured learning

2009 ◽  
Vol 77 (2-3) ◽  
pp. 195-224 ◽  
Author(s):  
Chun-Nan Hsu ◽  
Han-Shen Huang ◽  
Yu-Ming Chang ◽  
Yuh-Jye Lee
Keyword(s):  
Gradient Descent ◽  
Second Order ◽  
Structured Learning ◽  
Step Size ◽  
Single Pass ◽  
On Line
Sign in / Sign up

Export Citation Format

Share Document