scholarly journals LM-CMA: An Alternative to L-BFGS for Large-Scale Black Box Optimization

2017 ◽  
Vol 25 (1) ◽  
pp. 143-171 ◽  
Author(s):  
Ilya Loshchilov
Keyword(s):  
Covariance Matrix ◽  
Large Scale ◽  
Optimization Problems ◽  
Finite Difference Methods ◽  
Continuous Optimization ◽  
Black Box ◽  
Limited Memory ◽  
Covariance Matrix Adaptation ◽  
Comparable Performance ◽  
Dimension Invariance

Limited-memory BFGS (L-BFGS; Liu and Nocedal, 1989 ) is often considered to be the method of choice for continuous optimization when first- or second-order information is available. However, the use of L-BFGS can be complicated in a black box scenario where gradient information is not available and therefore should be numerically estimated. The accuracy of this estimation, obtained by finite difference methods, is often problem-dependent and may lead to premature convergence of the algorithm. This article demonstrates an alternative to L-BFGS, the limited memory covariance matrix adaptation evolution strategy (LM-CMA) proposed by Loshchilov ( 2014 ). LM-CMA is a stochastic derivative-free algorithm for numerical optimization of nonlinear, nonconvex optimization problems. Inspired by L-BFGS, LM-CMA samples candidate solutions according to a covariance matrix reproduced from m direction vectors selected during the optimization process. The decomposition of the covariance matrix into Cholesky factors allows reducing the memory complexity to [Formula: see text], where n is the number of decision variables. The time complexity of sampling one candidate solution is also [Formula: see text] but scales as only about 25 scalar-vector multiplications in practice. The algorithm has an important property of invariance with respect to strictly increasing transformations of the objective function; such transformations do not compromise its ability to approach the optimum. LM-CMA outperforms the original CMA-ES and its large-scale versions on nonseparable ill-conditioned problems with a factor increasing with problem dimension. Invariance properties of the algorithm do not prevent it from demonstrating a comparable performance to L-BFGS on nontrivial large-scale smooth and nonsmooth optimization problems.

Fast Covariance Matrix Adaptation for Large-Scale Black-Box Optimization

2020 ◽  
Vol 50 (5) ◽  
pp. 2073-2083 ◽  
Author(s):  
Zhenhua Li ◽  
Qingfu Zhang ◽  
Xi Lin ◽  
Hui-Ling Zhen
Keyword(s):  
Covariance Matrix ◽  
Large Scale ◽  
Black Box ◽  
Covariance Matrix Adaptation

Optimization Design of Trusses Based on Covariance Matrix Adaptation Evolution Strategy Algorithm

2012 ◽  
Vol 215-216 ◽  
pp. 133-137
Author(s):  
Guo Shao Su ◽  
Yan Zhang ◽  
Zhen Xing Wu ◽  
Liu Bin Yan
Keyword(s):  
Stochastic Optimization ◽  
Covariance Matrix ◽  
Optimization Problems ◽  
Fitness Function ◽  
Optimization Methods ◽  
Evolution Strategy ◽  
Covariance Matrix Adaptation ◽  
Study Results ◽  
Highly Nonlinear ◽  
Adaptation Evolution

Covariance matrix adaptation evolution strategy algorithm (CMA-ES) is a newly evolution algorithm. It has become a powerful tool for solving highly nonlinear multi-peak optimization problems. In many real-world optimization problems, the location of multiple optima is often required in a search space. In order to evaluate the solution, thousands of fitness function evaluations are involved that is a time consuming or expensive processes. Therefore, conventional stochastic optimization methods meet a special challenge for a very large number of problem function evaluations. Aiming to overcome the shortcoming of stochastic optimization methods in the high calculation cost, a truss optimal method based on CMA-ES algorithm is proposed and applied to solve the section and shape optimization problems of trusses. The study results show that the method is feasible and has the advantages of high accuracy, high efficiency and easy implementation.

scholarly journals Mirrored Orthogonal Sampling for Covariance Matrix Adaptation Evolution Strategies

2019 ◽  
Vol 27 (4) ◽  
pp. 699-725 ◽  
Author(s):  
Hao Wang ◽  
Michael Emmerich ◽  
Thomas Bäck
Keyword(s):  
Convergence Rate ◽  
Covariance Matrix ◽  
Sampling Technique ◽  
Black Box ◽  
Evolution Strategies ◽  
High Dimensional ◽  
Major Purpose ◽  
Search Spaces ◽  
Covariance Matrix Adaptation ◽  
Adaptation Evolution

Generating more evenly distributed samples in high dimensional search spaces is the major purpose of the recently proposed mirrored sampling technique for evolution strategies. The diversity of the mutation samples is enlarged and the convergence rate is therefore improved by the mirrored sampling. Motivated by the mirrored sampling technique, this article introduces a new derandomized sampling technique called mirrored orthogonal sampling. The performance of this new technique is both theoretically analyzed and empirically studied on the sphere function. In particular, the mirrored orthogonal sampling technique is applied to the well-known Covariance Matrix Adaptation Evolution Strategy (CMA-ES). The resulting algorithm is experimentally tested on the well-known Black-Box Optimization Benchmark (BBOB). By comparing the results from the benchmark, mirrored orthogonal sampling is found to outperform both the standard CMA-ES and its variant using mirrored sampling.

scholarly journals Black-box combinatorial optimization using models with integer-valued minima

2020 ◽  
Author(s):  
Laurens Bliek ◽  
Sicco Verwer ◽  
Mathijs de Weerdt
Keyword(s):  
Optimization Algorithm ◽  
Surrogate Model ◽  
Optimization Problems ◽  
Random Search ◽  
Continuous Optimization ◽  
Optimal Solution ◽  
Black Box ◽  
Combinatorial Problems ◽  
Bayesian Optimization ◽  
Bayesian Optimization Algorithm

Abstract When a black-box optimization objective can only be evaluated with costly or noisy measurements, most standard optimization algorithms are unsuited to find the optimal solution. Specialized algorithms that deal with exactly this situation make use of surrogate models. These models are usually continuous and smooth, which is beneficial for continuous optimization problems, but not necessarily for combinatorial problems. However, by choosing the basis functions of the surrogate model in a certain way, we show that it can be guaranteed that the optimal solution of the surrogate model is integer. This approach outperforms random search, simulated annealing and a Bayesian optimization algorithm on the problem of finding robust routes for a noise-perturbed traveling salesman benchmark problem, with similar performance as another Bayesian optimization algorithm, and outperforms all compared algorithms on a convex binary optimization problem with a large number of variables.

scholarly journals Fast and reliable transient simulation and continuous optimization of large-scale gas networks

2022 ◽  
Author(s):  
Pia Domschke ◽  
Oliver Kolb ◽  
Jens Lang
Keyword(s):  
Large Scale ◽  
Gas Flow ◽  
Optimization Problems ◽  
Continuous Optimization ◽  
Gas Pipeline ◽  
Physical Models ◽  
Active Set ◽  
Simulation And Optimization ◽  
Pipeline Systems ◽  
Adjoint Approach

AbstractWe are concerned with the simulation and optimization of large-scale gas pipeline systems in an error-controlled environment. The gas flow dynamics is locally approximated by sufficiently accurate physical models taken from a hierarchy of decreasing complexity and varying over time. Feasible work regions of compressor stations consisting of several turbo compressors are included by semiconvex approximations of aggregated characteristic fields. A discrete adjoint approach within a first-discretize-then-optimize strategy is proposed and a sequential quadratic programming with an active set strategy is applied to solve the nonlinear constrained optimization problems resulting from a validation of nominations. The method proposed here accelerates the computation of near-term forecasts of sudden changes in the gas management and allows for an economic control of intra-day gas flow schedules in large networks. Case studies for real gas pipeline systems show the remarkable performance of the new method.

scholarly journals Tree-Seed Algorithm for Large-Scale Binary Optimization

2018 ◽  
Vol 3 (1) ◽  
pp. 48 ◽  
Author(s):  
Ahmet Cevahir Cinar ◽  
Hazim Iscan ◽  
Mustafa Servet Kiran
Keyword(s):  
Large Scale ◽  
Optimization Problems ◽  
Logic Gate ◽  
Continuous Optimization ◽  
Population Based ◽  
Benchmark Problems ◽  
Binary Optimization ◽  
Continuous Optimization Problems ◽  
Computation Algorithms ◽  
Tree Seed

Population-based swarm or evolutionary computation algorithms in optimization are attracted the interest of the researchers due their simple structure, optimization performance, easy-adaptation. Binary optimization problems can be also solved by using these algorithms. This paper focuses on solving large scale binary optimization problems by using Tree-Seed Algorithm (TSA) proposed for solving continuous optimization problems by imitating relationship between the trees and their seeds in nature. The basic TSA is modified by using xor logic gate for solving binary optimization problems in this study. In order to investigate the performance of the proposed algorithm, the numeric benchmark problems with the different dimensions are considered and obtained results show that the proposed algorithm produces effective and comparable solutions in terms of solution quality.Keywords: binary optimization, tree-seed algorithm, xor-gate, large-scale optimization

scholarly journals Implementing and modifying Broyden class updates for large scale optimization

2020 ◽  
Author(s):  
Martin Buhmann ◽  
Dirk Siegel
Keyword(s):  
Large Scale ◽  
Optimization Problems ◽  
Substantial Improvement ◽  
Gradient Algorithm ◽  
Large Scale Optimization ◽  
Fixed Amount ◽  
Limited Memory ◽  
Broyden Class ◽  
Scale Optimization ◽  
Number Of Iterations

Abstract We consider Broyden class updates for large scale optimization problems in n dimensions, restricting attention to the case when the initial second derivative approximation is the identity matrix. Under this assumption we present an implementation of the Broyden class based on a coordinate transformation on each iteration. It requires only $$2nk + O(k^{2}) + O(n)$$ 2 n k + O ( k 2 ) + O ( n ) multiplications on the kth iteration and stores $$nK+ O(K^2) + O(n)$$ n K + O ( K 2 ) + O ( n ) numbers, where K is the total number of iterations. We investigate a modification of this algorithm by a scaling approach and show a substantial improvement in performance over the BFGS method. We also study several adaptations of the new implementation to the limited memory situation, presenting algorithms that work with a fixed amount of storage independent of the number of iterations. We show that one such algorithm retains the property of quadratic termination. The practical performance of the new methods is compared with the performance of Nocedal’s (Math Comput 35:773--782, 1980) method, which is considered the benchmark in limited memory algorithms. The tests show that the new algorithms can be significantly more efficient than Nocedal’s method. Finally, we show how a scaling technique can significantly improve both Nocedal’s method and the new generalized conjugate gradient algorithm.

A Modified Limited Memory BFGS Method for Non-Convex Minimization

2014 ◽  
Vol 530-531 ◽  
pp. 367-371
Author(s):  
Ting Feng Li ◽  
Yu Ting Zhang ◽  
Sheng Hui Yan
Keyword(s):  
Large Scale ◽  
Optimization Problems ◽  
Test Problems ◽  
Bfgs Method ◽  
Remarkable Feature ◽  
Limited Memory Bfgs Method ◽  
Limited Memory ◽  
Unconstrained Optimization Problems ◽  
Global Convergence Property ◽  
Large Scale Unconstrained Optimization

In this paper, a modified limited memory BFGS method for solving large-scale unconstrained optimization problems is proposed. A remarkable feature of the proposed method is that it possesses a global convergence property even without convexity assumption on the objective function. The implementations of the algorithm on CUTE test problems are reported, which suggest that a slight improvement has been achieved.

Sign in / Sign up

Export Citation Format

Share Document