Derivative-free optimization for expensive constrained problems using a novel expected improvement objective function

AIChE Journal ◽  
2014 ◽  
Vol 60 (7) ◽  
pp. 2462-2474 ◽  
Author(s):  
Fani Boukouvala ◽  
Marianthi G. Ierapetritou
Keyword(s):  
Objective Function ◽  
Expected Improvement ◽  
Constrained Problems ◽  
Derivative Free Optimization ◽  
Derivative Free

scholarly journals Derivative-free optimization and filter methods to solve nonlinear constrained problems

2009 ◽  
Vol 86 (10-11) ◽  
pp. 1841-1851 ◽  
Author(s):  
Aldina Correia ◽  
João Matias ◽  
Pedro Mestre ◽  
Carlos Serôdio
Keyword(s):  
Constrained Problems ◽  
Filter Methods ◽  
Derivative Free Optimization ◽  
Derivative Free

scholarly journals Randomized Simplicial Hessian Update

Mathematics ◽  
2021 ◽  
Vol 9 (15) ◽  
pp. 1775
Author(s):  
Árpád Bűrmen ◽  
Tadej Tuma ◽  
Jernej Olenšek
Keyword(s):  
Lower Bound ◽  
Dimensional Subspace ◽  
Expected Improvement ◽  
Closed Form Expression ◽  
Frobenius Norm ◽  
Derivative Free Optimization ◽  
Derivative Free ◽  
Derivative Information ◽  
Approximate Hessian ◽  
Special Case

Recently, a derivative-free optimization algorithm was proposed that utilizes a minimum Frobenius norm (MFN) Hessian update for estimating the second derivative information, which in turn is used for accelerating the search. The proposed update formula relies only on computed function values and is a closed-form expression for a special case of a more general approach first published by Powell. This paper analyzes the convergence of the proposed update formula under the assumption that the points from Rn where the function value is known are random. The analysis assumes that the N+2 points used by the update formula are obtained by adding N+1 vectors to a central point. The vectors are obtained by transforming a prototype set of N+1 vectors with a random orthogonal matrix from the Haar measure. The prototype set must positively span a N≤n dimensional subspace. Because the update is random by nature we can estimate a lower bound on the expected improvement of the approximate Hessian. This lower bound was derived for a special case of the proposed update by Leventhal and Lewis. We generalize their result and show that the amount of improvement greatly depends on N as well as the choice of the vectors in the prototype set. The obtained result is then used for analyzing the performance of the update based on various commonly used prototype sets. One of the results obtained by this analysis states that a regular n-simplex is a bad choice for a prototype set because it does not guarantee any improvement of the approximate Hessian.

scholarly journals Optimal Wells Placement to Maximize the Field Coverage Using Derivative-Free Optimization

2020 ◽  
Vol 178 ◽  
pp. 65-74
Author(s):  
Ksenia Balabaeva ◽  
Liya Akmadieva ◽  
Sergey Kovalchuk
Keyword(s):  
Derivative Free Optimization ◽  
Derivative Free

Benchmarking and Field-Testing of the Distributed Quasi-Newton Derivative-Free Optimization Method for Field Development Optimization

2021 ◽  
Author(s):  
Faruk Alpak ◽  
Yixuan Wang ◽  
Guohua Gao ◽  
Vivek Jain
Keyword(s):  
Optimization Problems ◽  
Field Testing ◽  
Optimization Method ◽  
Training Data ◽  
Local Optima ◽  
Field Development ◽  
Derivative Free Optimization ◽  
Derivative Free ◽  
Data Points ◽  
Quasi Newton

Abstract Recently, a novel distributed quasi-Newton (DQN) derivative-free optimization (DFO) method was developed for generic reservoir performance optimization problems including well-location optimization (WLO) and well-control optimization (WCO). DQN is designed to effectively locate multiple local optima of highly nonlinear optimization problems. However, its performance has neither been validated by realistic applications nor compared to other DFO methods. We have integrated DQN into a versatile field-development optimization platform designed specifically for iterative workflows enabled through distributed-parallel flow simulations. DQN is benchmarked against alternative DFO techniques, namely, the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method hybridized with Direct Pattern Search (BFGS-DPS), Mesh Adaptive Direct Search (MADS), Particle Swarm Optimization (PSO), and Genetic Algorithm (GA). DQN is a multi-thread optimization method that distributes an ensemble of optimization tasks among multiple high-performance-computing nodes. Thus, it can locate multiple optima of the objective function in parallel within a single run. Simulation results computed from one DQN optimization thread are shared with others by updating a unified set of training data points composed of responses (implicit variables) of all successful simulation jobs. The sensitivity matrix at the current best solution of each optimization thread is approximated by a linear-interpolation technique using all or a subset of training-data points. The gradient of the objective function is analytically computed using the estimated sensitivities of implicit variables with respect to explicit variables. The Hessian matrix is then updated using the quasi-Newton method. A new search point for each thread is solved from a trust-region subproblem for the next iteration. In contrast, other DFO methods rely on a single-thread optimization paradigm that can only locate a single optimum. To locate multiple optima, one must repeat the same optimization process multiple times starting from different initial guesses for such methods. Moreover, simulation results generated from a single-thread optimization task cannot be shared with other tasks. Benchmarking results are presented for synthetic yet challenging WLO and WCO problems. Finally, DQN method is field-tested on two realistic applications. DQN identifies the global optimum with the least number of simulations and the shortest run time on a synthetic problem with known solution. On other benchmarking problems without a known solution, DQN identified compatible local optima with reasonably smaller numbers of simulations compared to alternative techniques. Field-testing results reinforce the auspicious computational attributes of DQN. Overall, the results indicate that DQN is a novel and effective parallel algorithm for field-scale development optimization problems.

scholarly journals Optimization based Layer-wise Magnitude-based Pruning for DNN Compression

2018 ◽  
Author(s):  
Guiying Li ◽  
Chao Qian ◽  
Chunhui Jiang ◽  
Xiaofen Lu ◽  
Ke Tang
Keyword(s):  
Deep Neural Network ◽  
Optimization Problem ◽  
State Of The Art ◽  
Automatic Tuning ◽  
Constrained Optimization Problem ◽  
Derivative Free Optimization ◽  
Derivative Free ◽  
Threshold Tuning ◽  
Model Subject ◽  
Pruning Methods

Layer-wise magnitude-based pruning (LMP) is a very popular method for deep neural network (DNN) compression. However, tuning the layer-specific thresholds is a difficult task, since the space of threshold candidates is exponentially large and the evaluation is very expensive. Previous methods are mainly by hand and require expertise. In this paper, we propose an automatic tuning approach based on optimization, named OLMP. The idea is to transform the threshold tuning problem into a constrained optimization problem (i.e., minimizing the size of the pruned model subject to a constraint on the accuracy loss), and then use powerful derivative-free optimization algorithms to solve it. To compress a trained DNN, OLMP is conducted within a new iterative pruning and adjusting pipeline. Empirical results show that OLMP can achieve the best pruning ratio on LeNet-style models (i.e., 114 times for LeNet-300-100 and 298 times for LeNet-5) compared with some state-of-the- art DNN pruning methods, and can reduce the size of an AlexNet-style network up to 82 times without accuracy loss.

Sign in / Sign up

Export Citation Format

Share Document