scholarly journals An Effective Optimization Method for Machine Learning Based on ADAM

2020 ◽  
Vol 10 (3) ◽  
pp. 1073 ◽  
Author(s):  
Dokkyun Yi ◽  
Jaehyun Ahn ◽  
Sangmin Ji
Keyword(s):  
Machine Learning ◽  
Cost Function ◽  
Gradient Descent ◽  
Local Minimum ◽  
Optimization Method ◽  
Activation Function ◽  
Numerical Comparison ◽  
First Derivative ◽  
Artificial Neural Network Ann ◽  
The Cost

A machine is taught by finding the minimum value of the cost function which is induced by learning data. Unfortunately, as the amount of learning increases, the non-liner activation function in the artificial neural network (ANN), the complexity of the artificial intelligence structures, and the cost function’s non-convex complexity all increase. We know that a non-convex function has local minimums, and that the first derivative of the cost function is zero at a local minimum. Therefore, the methods based on a gradient descent optimization do not undergo further change when they fall to a local minimum because they are based on the first derivative of the cost function. This paper introduces a novel optimization method to make machine learning more efficient. In other words, we construct an effective optimization method for non-convex cost function. The proposed method solves the problem of falling into a local minimum by adding the cost function in the parameter update rule of the ADAM method. We prove the convergence of the sequences generated from the proposed method and the superiority of the proposed method by numerical comparison with gradient descent (GD, ADAM, and AdaMax).

An Enhanced Optimization Scheme Based on Gradient Descent Methods for Machine Learning

Symmetry ◽  
2019 ◽  
Vol 11 (7) ◽  
pp. 942 ◽  
Author(s):  
Dokkyun Yi ◽  
Sangmin Ji ◽  
Sunyoung Bu
Keyword(s):  
Machine Learning ◽  
Cost Function ◽  
Gradient Descent ◽  
Local Minimum ◽  
Global Minimum ◽  
Convergence Condition ◽  
First Derivative ◽  
Estimation Scheme ◽  
The Cost ◽  
Learning Data

A The learning process of machine learning consists of finding values of unknown weights in a cost function by minimizing the cost function based on learning data. However, since the cost function is not convex, it is conundrum to find the minimum value of the cost function. The existing methods used to find the minimum values usually use the first derivative of the cost function. When even the local minimum (but not a global minimum) is reached, since the first derivative of the cost function becomes zero, the methods give the local minimum values, so that the desired global minimum cannot be found. To overcome this problem, in this paper we modified one of the existing schemes—the adaptive momentum estimation scheme—by adding a new term, so that it can prevent the new optimizer from staying at local minimum. The convergence condition for the proposed scheme and the convergence value are also analyzed, and further explained through several numerical experiments whose cost function is non-convex.

scholarly journals An Adaptive Optimization Method Based on Learning Rate Schedule for Neural Networks

2021 ◽  
Vol 11 (2) ◽  
pp. 850
Author(s):  
Dokkyun Yi ◽  
Sangmin Ji ◽  
Jieun Park
Keyword(s):  
Artificial Intelligence ◽  
Cost Function ◽  
Numerical Experiments ◽  
Global Minimum ◽  
Optimization Method ◽  
Learning Method ◽  
Adaptive Optimization ◽  
The Cost ◽  
Proof Of Convergence ◽  
Learning Data

Artificial intelligence (AI) is achieved by optimizing the cost function constructed from learning data. Changing the parameters in the cost function is an AI learning process (or AI learning for convenience). If AI learning is well performed, then the value of the cost function is the global minimum. In order to obtain the well-learned AI learning, the parameter should be no change in the value of the cost function at the global minimum. One useful optimization method is the momentum method; however, the momentum method has difficulty stopping the parameter when the value of the cost function satisfies the global minimum (non-stop problem). The proposed method is based on the momentum method. In order to solve the non-stop problem of the momentum method, we use the value of the cost function to our method. Therefore, as the learning method processes, the mechanism in our method reduces the amount of change in the parameter by the effect of the value of the cost function. We verified the method through proof of convergence and numerical experiments with existing methods to ensure that the learning works well.

scholarly journals Statistical and Machine Learning-Driven Optimization of Mechanical Properties in Designing Durable HDPE Nanobiocomposites

Polymers ◽  
2021 ◽  
Vol 13 (18) ◽  
pp. 3100
Author(s):  
Anusha Mairpady ◽  
Abdel-Hamid I. Mourad ◽  
Mohammad Sayem Mozumder
Keyword(s):  
Machine Learning ◽  
Mechanical Properties ◽  
Genetic Algorithm ◽  
Machine Learning Techniques ◽  
Hybrid Technique ◽  
Learning Techniques ◽  
Artificial Neural Network Ann ◽  
Major Factors ◽  
The Cost ◽  
Design Factors

The selection of nanofillers and compatibilizing agents, and their size and concentration, are always considered to be crucial in the design of durable nanobiocomposites with maximized mechanical properties (i.e., fracture strength (FS), yield strength (YS), Young’s modulus (YM), etc). Therefore, the statistical optimization of the key design factors has become extremely important to minimize the experimental runs and the cost involved. In this study, both statistical (i.e., analysis of variance (ANOVA) and response surface methodology (RSM)) and machine learning techniques (i.e., artificial intelligence-based techniques (i.e., artificial neural network (ANN) and genetic algorithm (GA)) were used to optimize the concentrations of nanofillers and compatibilizing agents of the injection-molded HDPE nanocomposites. Initially, through ANOVA, the concentrations of TiO2 and cellulose nanocrystals (CNCs) and their combinations were found to be the major factors in improving the durability of the HDPE nanocomposites. Further, the data were modeled and predicted using RSM, ANN, and their combination with a genetic algorithm (i.e., RSM-GA and ANN-GA). Later, to minimize the risk of local optimization, an ANN-GA hybrid technique was implemented in this study to optimize multiple responses, to develop the nonlinear relationship between the factors (i.e., the concentration of TiO2 and CNCs) and responses (i.e., FS, YS, and YM), with minimum error and with regression values above 95%.

Convergence of Simulated Annealing with Feedback Temperature Schedules

1997 ◽  
Vol 11 (3) ◽  
pp. 279-304 ◽  
Author(s):  
M. Kolonko ◽  
M. T. Tran
Keyword(s):  
Simulated Annealing ◽  
Cost Function ◽  
Job Shop ◽  
Job Shop Scheduling ◽  
Optimization Method ◽  
Search Process ◽  
Local Optimum ◽  
Fixed Sequence ◽  
Temperature Parameter ◽  
The Cost

It is well known that the standard simulated annealing optimization method converges in distribution to the minimum of the cost function if the probability a for accepting an increase in costs goes to 0. α is controlled by the “temperature” parameter, which in the standard setup is a fixed sequence of values converging slowly to 0. We study a more general model in which the temperature may depend on the state of the search process. This allows us to adapt the temperature to the landscape of the cost function. The temperature may temporarily rise such that the process can leave a local optimum more easily. We give weak conditions on the temperature schedules such that the process of solutions finally concentrates near the optimal solutions. We also briefly sketch computational results for the job shop scheduling problem.

scholarly journals Optimal Eco-Driving Cycles for Conventional Vehicles Using a Genetic Algorithm

Energies ◽  
2020 ◽  
Vol 13 (17) ◽  
pp. 4362
Author(s):  
Subramaniam Saravana Sankar ◽  
Yiqun Xia ◽  
Julaluk Carmai ◽  
Saiprasit Koetniyom
Keyword(s):  
Genetic Algorithm ◽  
Cost Function ◽  
Fuel Consumption ◽  
Optimization Algorithm ◽  
Internal Combustion Engines ◽  
Optimization Method ◽  
Computational Time ◽  
Weight Factor ◽  
Driving Cycles ◽  
The Cost

The goal of this work is to compute the eco-driving cycles for vehicles equipped with internal combustion engines by using a genetic algorithm (GA) with a focus on reducing energy consumption. The proposed GA-based optimization method uses an optimal control problem (OCP), which is framed considering both fuel consumption and driver comfort in the cost function formulation with the support of a tunable weight factor to enhance the overall performance of the algorithm. The results and functioning of the optimization algorithm are analyzed with several widely used standard driving cycles and a simulated real-world driving cycle. For the selected optimal weight factor, the simulation results show that an average reduction of eight percent in fuel consumption is achieved. The results of parallelization in computing the cost function indicates that the computational time required by the optimization algorithm is reduced based on the hardware used.

CHARACTERIZING ONE-LAYER ASSOCIATIVE NEURAL NETWORKS WITH OPTIMAL NOISE-REDUCTION ABILITY

1992 ◽  
Vol 06 (05) ◽  
pp. 1009-1025 ◽  
Author(s):  
TAO WANG ◽  
XIAOLIANG XING ◽  
XINHUA ZHUANG
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Cost Function ◽  
Noise Reduction ◽  
Gradient Descent ◽  
Storage Capacity ◽  
Learning Algorithm ◽  
Optimal Learning ◽  
The Neural Network ◽  
The Cost

In this paper, we describe an optimal learning algorithm for designing one-layer neural networks by means of global minimization. Taking the properties of a well-defined neural network into account, we derive a cost function to measure the goodness of the network quantitatively. The connection weights are determined by the gradient descent rule to minimize the cost function. The optimal learning algorithm is formed as either the unconstraint-based or the constraint-based minimization problem. It ensures the realization of each desired associative mapping with the best noise reduction ability in the sense of optimization. We also investigate the storage capacity of the neural network, the degree of noise reduction for a desired associative mapping, and the convergence of the learning algorithm in an analytic way. Finally, a large number of computer experimental results are presented.

scholarly journals On Generating Optimal Signal Probabilities for Random Tests: A Genetic Approach

VLSI Design ◽  
1996 ◽  
Vol 4 (3) ◽  
pp. 207-215 ◽  
Author(s):  
M. Srinivas ◽  
L. M. Patnaik
Keyword(s):  
Genetic Algorithms ◽  
Cost Function ◽  
Gradient Descent ◽  
Random Search ◽  
Search Space ◽  
Optimization Techniques ◽  
Gradient Descent Methods ◽  
Test Vectors ◽  
The Cost ◽  
Optimal Signal

Genetic Algorithms are robust search and optimization techniques. A Genetic Algorithm based approach for determining the optimal input distributions for generating random test vectors is proposed in the paper. A cost function based on the COP testability measure for determining the efficacy of the input distributions is discussed. A brief overview of Genetic Algorithms (GAs) and the specific details of our implementation are described. Experimental results based on ISCAS-85 benchmark circuits are presented. The performance of our GAbased approach is compared with previous results. While the GA generates more efficient input distributions than the previous methods which are based on gradient descent search, the overheads of the GA in computing the input distributions are larger.To account for the relatively quick convergence of the gradient descent methods, we analyze the landscape of the COP-based cost function. We prove that the cost function is unimodal in the search space. This feature makes the cost function amenable to optimization by gradient-descent techniques as compared to random search methods such as Genetic Algorithms.

scholarly journals Two-Dimensional Symmetric Box Delivery Motion Prediction and Validation: Subtask-Based Optimization Method

2020 ◽  
Vol 10 (24) ◽  
pp. 8798
Author(s):  
Yujiang Xiang ◽  
Shadman Tahmid ◽  
Paul Owens ◽  
James Yang
Keyword(s):  
Cost Function ◽  
Material Handling ◽  
Optimization Method ◽  
Joint Torque ◽  
Two Dimensional ◽  
Computationally Efficient ◽  
Inverse Dynamic ◽  
Manual Material Handling ◽  
Research Outcome ◽  
The Cost

Box delivery is a complicated manual material handling task which needs to consider the box weight, delivering speed, stability, and location. This paper presents a subtask-based inverse dynamic optimization formulation for determining the two-dimensional (2D) symmetric optimal box delivery motion. For the subtask-based formulation, the delivery task is divided into five subtasks: lifting, the first transition step, carrying, the second transition step, and unloading. To render a complete delivering task, each subtask is formulated as a separate optimization problem with appropriate boundary conditions. For carrying and lifting subtasks, the cost function is the sum of joint torque squared. In contrast, for transition subtasks, the cost function is the combination of joint discomfort and joint torque squared. Joint angle profiles are validated through experimental results using Pearson’s correlation coefficient (r) and root-mean-square-error (RMSE). Results show that the subtask-based approach is computationally efficient for complex box delivery motion simulation. This research outcome provides a practical guidance to prevent injury risks in joint torque space for workers who deliver heavy objects in their daily jobs.

A Novel Redundancy Resolution Method via Optimization with Additional Tasks

2013 ◽  
Vol 694-697 ◽  
pp. 1787-1792
Author(s):  
Xue Feng Zhou
Keyword(s):  
Cost Function ◽  
Optimization Method ◽  
Initial Configuration ◽  
Main Task ◽  
Joint Position ◽  
Redundancy Resolution ◽  
Redundant Manipulator ◽  
Resolution Method ◽  
Additional Task ◽  
The Cost

This paper presents an optimization method for redundant manipulator redundancy resolution with additional task. The cost function is a compromise between the requirement of accuracy of the main task, the accuracy of the additional task, and the feasibility of the joint velocities. The joint rates that minimize the cost function can be found, and the joint position trajectories can be integrated with an initial configuration. The effectiveness of this presented method is verified by a planar 3-DoF PRR manipulator.

scholarly journals Computational meta-heuristics based on Machine Learning to optimize fuel consumption of vessels using diesel engines

2021 ◽  
Vol 9 (5) ◽  
pp. 587-606
Author(s):  
Paulo Oliveira Siqueira Junior ◽  
Manoel Henrique Reis Nascimento ◽  
Ítalo Rodrigo Soares Silva ◽  
Ricardo Silva Parente ◽  
Milton Fonseca Júnior ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Genetic Algorithm ◽  
Decision Making ◽  
Training Algorithm ◽  
Fuel Cost ◽  
Artificial Neural Network Ann ◽  
The Cost ◽  
Quasi Newton ◽  
Optimize Fuel Consumption

With the expansion of means of river transportation, especially in the case of small and medium-sized vessels that make routes of greater distances, the cost of fuel, if not taken as an analysis criterion for a larger profit margin, is considered to be a primary factor , considering that the value of fuel specifically diesel to power internal combustion machines is high. Therefore, the use of tools that assist in decision-making becomes necessary, as is the case of the present research, which aims to contribute with a computational model of prediction and optimization of the best speed to decrease the fuel cost considering the characteristics of the SCANIA 315 machine. propulsion model, of a vessel from the river port of Manaus that carries out river transportation to several municipalities in Amazonas. According to the results of the simulations, the best training algorithm of the Artificial Neural Network (ANN) was the BFGS Quasi-Newton considering the characteristics of the engine for optimization with Genetic Algorithm (AG).

Sign in / Sign up

Export Citation Format

Share Document