scholarly journals Fractional-Order Deep Backpropagation Neural Network

2018 ◽  
Vol 2018 ◽  
pp. 1-10 ◽  
Author(s):  
Chunhui Bao ◽  
Yifei Pu ◽  
Yi Zhang
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Fractional Order ◽  
Bp Neural Network ◽  
Gradient Descent ◽  
Necessary Conditions ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Backpropagation Neural Network ◽  
Bp Neural Network Model

In recent years, the research of artificial neural networks based on fractional calculus has attracted much attention. In this paper, we proposed a fractional-order deep backpropagation (BP) neural network model with L2 regularization. The proposed network was optimized by the fractional gradient descent method with Caputo derivative. We also illustrated the necessary conditions for the convergence of the proposed network. The influence of L2 regularization on the convergence was analyzed with the fractional-order variational method. The experiments have been performed on the MNIST dataset to demonstrate that the proposed network was deterministically convergent and can effectively avoid overfitting.

scholarly journals IDENTIFICATION OF AREAS OF CORONAVIRUS COVID-19 INCIDENCE SPREADING BASED ON CLUSTER ANALYSIS METHOD

2021 ◽  
pp. 5-13
Author(s):  
Kseniia Bazilevych ◽  
Ievgen Meniailov ◽  
Dmytro Chumachenko
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Cluster Analysis ◽  
Data Analysis ◽  
Gradient Descent ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Adaptive Boosting ◽  
Software Product ◽  
Boosting Method

Subject: the use of the mathematical apparatus of neural networks for the scientific substantiation of anti-epidemic measures in order to reduce the incidence of diseases when making effective management decisions. Purpose: to apply cluster analysis, based on a neural network, to solve the problem of identifying areas of incidence. Tasks: to analyze methods of data analysis to solve the clustering problem; to develop a neural network method for clustering the territory of Ukraine according to the nature of the epidemic process COVID-19; on the basis of the developed method, to implement a data analysis software product to identify the areas of incidence of the disease using the example of the coronavirus COVID-19. Methods: models and methods of data analysis, models and methods of systems theory (based on the information approach), machine learning methods, in particular the Adaptive Boosting method (based on the gradient descent method), methods for training neural networks. Results: we used the data of the Center for Public Health of the Ministry of Health of Ukraine distributed over the regions of Ukraine on the incidence of COVID-19, the number of laboratory examined persons, the number of laboratory tests performed by PCR and ELISA methods, the number of laboratory tests of IgA, IgM, IgG; the model used data from March 2020 to December 2020, the modeling did not take into account data from the temporarily occupied territories of Ukraine; for cluster analysis, a neural network of 60 input neurons, 100 hidden neurons with an activation Fermi function and 4 output neurons was built; for the software implementation of the model, the programming language Python was used. Conclusions: analysis of methods for constructing neural networks; analysis of training methods for neural networks, including the use of the gradient descent method for the Adaptive Boosting method; all theoretical information described in this work was used to implement a software product for processing test data for COVID-19 in Ukraine; the division of the regions of Ukraine into zones of infection with the COVID-19 virus was carried out and a map of this division was presented.

Data-Driven Approach for Equipment Reliability Prediction Using Neural Network

2011 ◽  
Vol 411 ◽  
pp. 563-566 ◽  
Author(s):  
Feng Ding ◽  
Xing Ben Han
Keyword(s):  
Neural Network ◽  
Bp Neural Network ◽  
Gradient Descent ◽  
Optimization Theory ◽  
Descent Method ◽  
Numerical Control ◽  
Data Driven ◽  
Gradient Descent Method ◽  
Training Algorithm ◽  
Cnc Machine

BP neural network based data-driven method is proposed to predict reliability in this paper. The BP neural network prediction using Gradient Descent Method (GDM), Additional Momentum Gradient Descent Method (AMGDM) and Levenberg-Marquardt Method(L-M) based on numerical optimization theory of training algorithm are compared with different neuron number. The proposed approach is validated via age data collected from computer numerical control (CNC) machine tool in the field. The results from the proposed method show that perfect predicting performance is achieved under considering selecting suitable number of the hidden neurons and training algorithm. Remarks are outlined regarding the fact that BP neural network based on data-driven method is feasible, effective and adequate predicting accuracy can be obtained.

scholarly journals Predicting the Number of COVID-19 Sufferers in Malang City Using the Backpropagation Neural Network with the Fletcher–Reeves Method

2021 ◽  
Vol 2021 ◽  
pp. 1-9
Author(s):  
Syaiful Anam ◽  
Mochamad Hakim Akbar Assidiq Maulana ◽  
Noor Hidayat ◽  
Indah Yanti ◽  
Zuraidah Fitriah ◽  
...  
Keyword(s):  
Neural Network ◽  
Convergence Rate ◽  
Gradient Descent ◽  
Infected Individual ◽  
Prediction Method ◽  
Optimization Method ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Backpropagation Neural Network ◽  
Congenital Diseases

COVID-19 is a type of an infectious disease that is caused by the new coronavirus. The spread of COVID-19 needs to be suppressed because COVID-19 can cause death, especially for sufferers with congenital diseases and a weak immune system. COVID-19 spreads through direct contact, wherein the infected individual spreads the COVID-19 virus through cough, sneeze, or close contacts. Predicting the number of COVID-19 sufferers becomes an important task in the effort to curb the spread of COVID-19. Artificial neural network (ANN) is the prediction method that delivers effective results in doing this job. Backpropagation, a type of ANN algorithm, offers predictive problem solving with good performance. However, its performance depends on the optimization method applied during the training process. In general, the optimization method in ANN is the gradient descent method, which is known to have a slow convergence rate. Meanwhile, the Fletcher–Reeves method has a faster convergence rate than the gradient descent method. Based on this hypothesis, this paper proposes a prediction model for the number of COVID-19 sufferers in Malang using the Backpropagation neural network with the Fletcher–Reeves method. The experimental results show that the Backpropagation neural network with the Fletcher–Reeves method has a better performance than the Backpropagation neural network with the gradient descent method. This is shown by the Means Square Error (MSE) resulting from the proposed method which is smaller than the MSE resulting from the Backpropagation neural network with the gradient descent method.

scholarly journals Estimation of Navigation Mark Floating Based on Fractional-Order Gradient Descent with Momentum for RBF Neural Network

2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Qionglin Fang
Keyword(s):  
Neural Network ◽  
Fractional Order ◽  
Gradient Descent ◽  
Rbf Neural Network ◽  
Error Function ◽  
Initial Position ◽  
Descent Method ◽  
Local Optimum ◽  
Gradient Descent Method ◽  
The Neural Network

To address the difficulty of estimating the drift of the navigation marks, a fractional-order gradient with the momentum RBF neural network (FOGDM-RBF) is designed. The convergence is proved, and it is used to estimate the drifting trajectory of the navigation marks with different geographical locations. First, the weight of the neural network is set. The navigation mark’s meteorological, hydrological, and initial position data are taken as the input of the neural network. The neural network is trained and used to estimate the mark’s position. The navigation mark’s position is taken at a later time as the output of the neural network. The difference between the later position and the estimated position obtained from the neural network is the error function of the neural network. The influence of sea conditions and months are analyzed. The experimental results and error analysis show that FOGDM-RBF is better than other algorithms at trajectory estimation and interpolation, has better accuracy and generalization, and does not easily fall into the local optimum. It is effective at accelerating convergence speed and improving the performance of a gradient descent method.

scholarly journals Model Predictive Control of Nonlinear System Based on GA-RBP Neural Network and Improved Gradient Descent Method

Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-14
Author(s):  
Youming Wang ◽  
Didi Qing
Keyword(s):  
Neural Network ◽  
Model Predictive Control ◽  
Predictive Control ◽  
Bp Neural Network ◽  
Gradient Descent ◽  
Tracking Error ◽  
Hessian Matrix ◽  
Descent Method ◽  
Local Optimum ◽  
Gradient Descent Method

A model predictive control (MPC) method based on recursive backpropagation (RBP) neural network and genetic algorithm (GA) is proposed for a class of nonlinear systems with time delays and uncertainties. In the offline modeling stage, a multistep-ahead predictor with GA-RBP neural network is designed, where GA-BP neural network is used as a one-step prediction model and GA is employed to train the initial weights and bias of the BP neural network. The incorporation of GA into RBP can reduce the possibility of the BP neural network falling into a local optimum instead of reaching global optimization. In the online optimizing stage, a multistep-ahead GA-RBP neural network predictor and an improved gradient descent method (IGDM) are proposed to efficiently solve the online optimization problem of nonlinear MPC by minimizing a modified quadratic criterion. The designed MPC strategy can avoid information loss while linearizing the controlled system and computing the Hessian matrix and its inverse matrix. Experimental results show that the proposed approach can reduce the computational burden and improve the performance of MPC (i.e., the maximum overshoots, calculation time, rise time, and RMSE tracking error value) for the solution of nonlinear controlled systems.

scholarly journals SOME METHODS OF ADAPTIVE MULTILAYER NEURAL NETWORKS TRAINING

2014 ◽  
pp. 99-106
Author(s):  
Leonid Makhnist ◽  
Nikolaj Maniakov ◽  
Nikolaj Maniakov
Keyword(s):  
Neural Networks ◽  
Basic Concept ◽  
Gradient Descent ◽  
Descent Method ◽  
Gradient Descent Method ◽  
New Techniques ◽  
Adaptive Training ◽  
Multilayer Neural Networks

Is proposed two new techniques for multilayer neural networks training. Its basic concept is based on the gradient descent method. For every methodic are showed formulas for calculation of the adaptive training steps. Presented matrix algorithmizations for all of these techniques are very helpful in its program realization.

scholarly journals Real-Coded Quantum-Inspired Genetic Algorithm-Based BP Neural Network Algorithm

2015 ◽  
Vol 2015 ◽  
pp. 1-10 ◽  
Author(s):  
Jianyong Liu ◽  
Huaixiao Wang ◽  
Yangyang Sun ◽  
Chengqun Fu ◽  
Jie Guo
Keyword(s):  
Neural Network ◽  
Genetic Algorithm ◽  
Bp Neural Network ◽  
Search Space ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Quantum Genetic Algorithm ◽  
Binary Coding ◽  
Optimal Value ◽  
Neural Network Algorithm

The method that the real-coded quantum-inspired genetic algorithm (RQGA) used to optimize the weights and threshold of BP neural network is proposed to overcome the defect that the gradient descent method makes the algorithm easily fall into local optimal value in the learning process. Quantum genetic algorithm (QGA) is with good directional global optimization ability, but the conventional QGA is based on binary coding; the speed of calculation is reduced by the coding and decoding processes. So, RQGA is introduced to explore the search space, and the improved varied learning rate is adopted to train the BP neural network. Simulation test shows that the proposed algorithm is effective to rapidly converge to the solution conformed to constraint conditions.

scholarly journals Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode

2019 ◽  
Vol 9 (21) ◽  
pp. 4568
Author(s):  
Hyeyoung Park ◽  
Kwanyong Lee
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Learning Algorithm ◽  
Descent Method ◽  
Benchmark Problems ◽  
Stochastic Neural Networks ◽  
Gradient Descent Method ◽  
Natural Gradient ◽  
Convergence Properties ◽  
Data Set

Gradient descent method is an essential algorithm for learning of neural networks. Among diverse variations of gradient descent method that have been developed for accelerating learning speed, the natural gradient learning is based on the theory of information geometry on stochastic neuromanifold, and is known to have ideal convergence properties. Despite its theoretical advantages, the pure natural gradient has some limitations that prevent its practical usage. In order to get the explicit value of the natural gradient, it is required to know true probability distribution of input variables, and to calculate inverse of a matrix with the square size of the number of parameters. Though an adaptive estimation of the natural gradient has been proposed as a solution, it was originally developed for online learning mode, which is computationally inefficient for the learning of large data set. In this paper, we propose a novel adaptive natural gradient estimation for mini-batch learning mode, which is commonly adopted for big data analysis. For two representative stochastic neural network models, we present explicit rules of parameter updates and learning algorithm. Through experiments on three benchmark problems, we confirm that the proposed method has superior convergence properties to the conventional methods.

Sign in / Sign up

Export Citation Format

Share Document