scholarly journals A Three-Term Gradient Descent Method with Subspace Techniques

2021 ◽  
Vol 2021 ◽  
pp. 1-7
Author(s):  
Shengwei Yao ◽  
Yuping Wu ◽  
Jielan Yang ◽  
Jieqiong Xu
Keyword(s):  
Numerical Experiments ◽  
Gradient Descent ◽  
Line Search ◽  
Optimization Problems ◽  
Convergence Result ◽  
Descent Method ◽  
Search Direction ◽  
Gradient Descent Method ◽  
Approximation Model ◽  
Wolfe Line Search

We proposed a three-term gradient descent method that can be well applied to address the optimization problems in this article. The search direction of the obtained method is generated in a specific subspace. Specifically, a quadratic approximation model is applied in the process of generating the search direction. In order to reduce the amount of calculation and make the best use of the existing information, the subspace was made up of the gradient of the current and prior iteration point and the previous search direction. By using the subspace-based optimization technology, the global convergence result is established under Wolfe line search. The results of numerical experiments show that the new method is effective and robust.

scholarly journals Why Does Large Batch Training Result in Poor Generalization? A Comprehensive Explanation and a Better Strategy from the Viewpoint of Stochastic Optimization

2018 ◽  
Vol 30 (7) ◽  
pp. 2005-2023 ◽  
Author(s):  
Tomoumi Takase ◽  
Satoshi Oyama ◽  
Masahito Kurihara
Keyword(s):  
Gradient Descent ◽  
Optimization Problems ◽  
Descent Method ◽  
Batch Size ◽  
Gradient Descent Method ◽  
Neural Network Training ◽  
Nonconvex Optimization Problems ◽  
Large Batch ◽  
Network Training ◽  
Comprehensive Framework

We present a comprehensive framework of search methods, such as simulated annealing and batch training, for solving nonconvex optimization problems. These methods search a wider range by gradually decreasing the randomness added to the standard gradient descent method. The formulation that we define on the basis of this framework can be directly applied to neural network training. This produces an effective approach that gradually increases batch size during training. We also explain why large batch training degrades generalization performance, which previous studies have not clarified.

INITIAL IMPROVEMENT OF THE HYBRID ACCELERATED GRADIENT DESCENT PROCESS

2018 ◽  
Vol 98 (2) ◽  
pp. 331-338 ◽  
Author(s):  
STEFAN PANIĆ ◽  
MILENA J. PETROVIĆ ◽  
MIROSLAVA MIHAJLOV CAREVIĆ
Keyword(s):  
Gradient Descent ◽  
Line Search ◽  
Initial Step ◽  
Step Length ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Convergence Properties ◽  
Backtracking Line Search ◽  
Initial Improvement ◽  
Accelerated Gradient

We improve the convergence properties of the iterative scheme for solving unconstrained optimisation problems introduced in Petrovic et al. [‘Hybridization of accelerated gradient descent method’, Numer. Algorithms (2017), doi:10.1007/s11075-017-0460-4] by optimising the value of the initial step length parameter in the backtracking line search procedure. We prove the validity of the algorithm and illustrate its advantages by numerical experiments and comparisons.

scholarly journals A Mixed Spectral CD-DY Conjugate Gradient Method

2012 ◽  
Vol 2012 ◽  
pp. 1-10 ◽  
Author(s):  
Liu Jinkui ◽  
Du Xianglin ◽  
Wang Kairong
Keyword(s):  
Conjugate Gradient Method ◽  
Conjugate Gradient ◽  
Gradient Method ◽  
Line Search ◽  
Optimization Problems ◽  
Descent Method ◽  
Unconstrained Optimization Problems ◽  
Global Convergence Property ◽  
Wolfe Line Search ◽  
Spectral Conjugate Gradient

A mixed spectral CD-DY conjugate descent method for solving unconstrained optimization problems is proposed, which combines the advantages of the spectral conjugate gradient method, the CD method, and the DY method. Under the Wolfe line search, the proposed method can generate a descent direction in each iteration, and the global convergence property can be also guaranteed. Numerical results show that the new method is efficient and stationary compared to the CD (Fletcher 1987) method, the DY (Dai and Yuan 1999) method, and the SFR (Du and Chen 2008) method; so it can be widely used in scientific computation.

scholarly journals SSGD: A Safe and Efficient Method of Gradient Descent

2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Jinhuan Duan ◽  
Xianxian Li ◽  
Shiqi Gao ◽  
Zili Zhong ◽  
Jinyan Wang
Keyword(s):  
Gradient Descent ◽  
Large Scale ◽  
Optimization Problems ◽  
Unit Vector ◽  
Descent Method ◽  
Stochastic Gradient ◽  
Learning System ◽  
Training Data ◽  
Stochastic Gradient Descent ◽  
Gradient Descent Method

With the vigorous development of artificial intelligence technology, various engineering technology applications have been implemented one after another. The gradient descent method plays an important role in solving various optimization problems, due to its simple structure, good stability, and easy implementation. However, in multinode machine learning system, the gradients usually need to be shared, which will cause privacy leakage, because attackers can infer training data with the gradient information. In this paper, to prevent gradient leakage while keeping the accuracy of the model, we propose the super stochastic gradient descent approach to update parameters by concealing the modulus length of gradient vectors and converting it or them into a unit vector. Furthermore, we analyze the security of super stochastic gradient descent approach and demonstrate that our algorithm can defend against the attacks on the gradient. Experiment results show that our approach is obviously superior to prevalent gradient descent approaches in terms of accuracy, robustness, and adaptability to large-scale batches. Interestingly, our algorithm can also resist model poisoning attacks to a certain extent.

scholarly journals Multiplicative parameters in gradient descent methods

Filomat ◽  
2009 ◽  
Vol 23 (3) ◽  
pp. 23-36 ◽  
Author(s):  
Predrag Stanimirovic ◽  
Marko Miladinovic ◽  
Snezana Djordjevic
Keyword(s):  
Gradient Descent ◽  
Line Search ◽  
Main Idea ◽  
Step Length ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Calculation Algorithm ◽  
Modified Newton Method ◽  
Backtracking Line Search ◽  
Algorithm Construction

We introduced an algorithm for unconstrained optimization based on the reduction of the modified Newton method with line search into a gradient descent method. Main idea used in the algorithm construction is approximation of Hessian by a diagonal matrix. The step length calculation algorithm is based on the Taylor's development in two successive iterative points and the backtracking line search procedure.

ON MINIMIZATION OF IMPACT OF MAGNETIC ELEMENTS ADJUSTMENT TOLERANCES ON BEAM ORBIT DEVIATIONS IN SYNCHROTRONS

2021 ◽  
Author(s):  
Владислав Владимирвоич Алцыбеев
Keyword(s):  
Numerical Experiments ◽  
Gradient Descent ◽  
Charged Particles ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Alignment Error ◽  
Magnetic Elements ◽  
Beam Orbit

Рассматривается задача минимазации отклонений орбиты пучка заряженных частиц в синхротронах, вызванной погрешностью юстировки квадруполей. Разработан метод оптимизации траектории орбиты, основанный на применении роевых вычислений и метода градиентного спуска. Приведены результаты численных экспериментов. The problem of minimizing the deviations of the orbit of a beam of charged particles in synchrotrons caused by the alignment error of the quadrupoles is considered. A method for optimizing the orbit trajectory based on the use of swarm computations and the gradient descent method has been developed. The results of numerical experiments are presented.

scholarly journals A Distributed Conjugate Gradient Online Learning Method over Networks

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-13
Author(s):  
Cuixia Xu ◽  
Junlong Zhu ◽  
Youlin Shang ◽  
Qingtao Wu
Keyword(s):  
Conjugate Gradient ◽  
Gradient Descent ◽  
Line Search ◽  
Descent Method ◽  
Convergence Speed ◽  
Gradient Algorithm ◽  
Gradient Descent Method ◽  
Objective Functions ◽  
Regret Bound ◽  
Number Of Iterations

In a distributed online optimization problem with a convex constrained set over an undirected multiagent network, the local objective functions are convex and vary over time. Most of the existing methods used to solve this problem are based on the fastest gradient descent method. However, the convergence speed of these methods is decreased with an increase in the number of iterations. To accelerate the convergence speed of the algorithm, we present a distributed online conjugate gradient algorithm, different from a gradient method, in which the search directions are a set of vectors that are conjugated to each other and the step sizes are obtained through an accurate line search. We analyzed the convergence of the algorithm theoretically and obtained a regret bound of OT, where T is the number of iterations. Finally, numerical experiments conducted on a sensor network demonstrate the performance of the proposed algorithm.

scholarly journals Global Convergence of a Modified LS Method

2012 ◽  
Vol 2012 ◽  
pp. 1-9
Author(s):  
Liu JinKui ◽  
Du Xianglin
Keyword(s):  
Global Convergence ◽  
Numerical Experiments ◽  
Line Search ◽  
Optimization Problems ◽  
Gradient Methods ◽  
Conjugate Gradient Methods ◽  
Uniformly Convex ◽  
Unconstrained Optimization Problems ◽  
Strong Wolfe Line Search ◽  
Wolfe Line Search

The LS method is one of the effective conjugate gradient methods in solving the unconstrained optimization problems. The paper presents a modified LS method on the basis of the famous LS method and proves the strong global convergence for the uniformly convex functions and the global convergence for general functions under the strong Wolfe line search. The numerical experiments show that the modified LS method is very effective in practice.

scholarly journals Optimizing a polynomial function on a quantum processor

2021 ◽  
Vol 7 (1) ◽  
Author(s):  
Keren Li ◽  
Shijie Wei ◽  
Pan Gao ◽  
Feihao Zhang ◽  
Zengrong Zhou ◽  
...  
Keyword(s):  
Gradient Descent ◽  
Local Minimum ◽  
Data Science ◽  
Optimization Problems ◽  
Descent Method ◽  
Machine Learning Algorithms ◽  
High Dimensional ◽  
Gradient Descent Method ◽  
Quantum Processor ◽  
Computational Resources

AbstractThe gradient descent method is central to numerical optimization and is the key ingredient in many machine learning algorithms. It promises to find a local minimum of a function by iteratively moving along the direction of the steepest descent. Since for high-dimensional problems the required computational resources can be prohibitive, it is desirable to investigate quantum versions of the gradient descent, such as the recently proposed (Rebentrost et al.1). Here, we develop this protocol and implement it on a quantum processor with limited resources. A prototypical experiment is shown with a four-qubit nuclear magnetic resonance quantum processor, which demonstrates the iterative optimization process. Experimentally, the final point converged to the local minimum with a fidelity >94%, quantified via full-state tomography. Moreover, our method can be employed to a multidimensional scaling problem, showing the potential to outperform its classical counterparts. Considering the ongoing efforts in quantum information and data science, our work may provide a faster approach to solving high-dimensional optimization problems and a subroutine for future practical quantum computers.

Sign in / Sign up

Export Citation Format

Share Document