Training algorithms for backpropagation neural networks with optimal descent factor

1990 ◽  
Vol 26 (20) ◽  
pp. 1698 ◽  
Author(s):  
X-H. Yu ◽  
S-X. Cheng
Keyword(s):  
Neural Networks ◽  
Training Algorithms ◽  
Backpropagation Neural Networks

scholarly journals Neural Network Approach for Global Solar Irradiance Prediction at Extremely Short-Time-Intervals Using Particle Swarm Optimization Algorithm

Energies ◽  
2021 ◽  
Vol 14 (4) ◽  
pp. 1213
Author(s):  
Ahmed Aljanad ◽  
Nadia M. L. Tan ◽  
Vassilios G. Agelidis ◽  
Hussain Shareef
Keyword(s):  
Neural Networks ◽  
Particle Swarm Optimization ◽  
Solar Irradiance ◽  
Particle Swarm ◽  
Time Interval ◽  
Time Intervals ◽  
Swarm Optimization ◽  
Backpropagation Neural Networks ◽  
Proposed Model ◽  
Short Time

Hourly global solar irradiance (GSR) data are required for sizing, planning, and modeling of solar photovoltaic farms. However, operating and controlling such farms exposed to varying environmental conditions, such as fast passing clouds, necessitates GSR data to be available for very short time intervals. Classical backpropagation neural networks do not perform satisfactorily when predicting parameters within short intervals. This paper proposes a hybrid backpropagation neural networks based on particle swarm optimization. The particle swarm algorithm is used as an optimization algorithm within the backpropagation neural networks to optimize the number of hidden layers and neurons used and its learning rate. The proposed model can be used as a reliable model in predicting changes in the solar irradiance during short time interval in tropical regions such as Malaysia and other regions. Actual global solar irradiance data of 5-s and 1-min intervals, recorded by weather stations, are applied to train and test the proposed algorithm. Moreover, to ensure the adaptability and robustness of the proposed technique, two different cases are evaluated using 1-day and 3-days profiles, for two different time intervals of 1-min and 5-s each. A set of statistical error indices have been introduced to evaluate the performance of the proposed algorithm. From the results obtained, the 3-days profile’s performance evaluation of the BPNN-PSO are 1.7078 of RMSE, 0.7537 of MAE, 0.0292 of MSE, and 31.4348 of MAPE (%), at 5-s time interval, where the obtained results of 1-min interval are 0.6566 of RMSE, 0.2754 of MAE, 0.0043 of MSE, and 1.4732 of MAPE (%). The results revealed that proposed model outperformed the standalone backpropagation neural networks method in predicting global solar irradiance values for extremely short-time intervals. In addition to that, the proposed model exhibited high level of predictability compared to other existing models.

scholarly journals Trigonometric Inference Providing Learning in Deep Neural Networks

2021 ◽  
Vol 11 (15) ◽  
pp. 6704
Author(s):  
Jingyong Cai ◽  
Masashi Takemoto ◽  
Yuming Qiu ◽  
Hironori Nakajo
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Trigonometric Approximation ◽  
Model Parameters ◽  
Training Algorithms ◽  
Activation Functions ◽  
Classical Training ◽  
Sum Formula

Despite being heavily used in the training of deep neural networks (DNNs), multipliers are resource-intensive and insufficient in many different scenarios. Previous discoveries have revealed the superiority when activation functions, such as the sigmoid, are calculated by shift-and-add operations, although they fail to remove multiplications in training altogether. In this paper, we propose an innovative approach that can convert all multiplications in the forward and backward inferences of DNNs into shift-and-add operations. Because the model parameters and backpropagated errors of a large DNN model are typically clustered around zero, these values can be approximated by their sine values. Multiplications between the weights and error signals are transferred to multiplications of their sine values, which are replaceable with simpler operations with the help of the product to sum formula. In addition, a rectified sine activation function is utilized for further converting layer inputs into sine values. In this way, the original multiplication-intensive operations can be computed through simple add-and-shift operations. This trigonometric approximation method provides an efficient training and inference alternative for devices with insufficient hardware multipliers. Experimental results demonstrate that this method is able to obtain a performance close to that of classical training algorithms. The approach we propose sheds new light on future hardware customization research for machine learning.

scholarly journals A Review of Algorithms and Hardware Implementations for Spiking Neural Networks

2021 ◽  
Vol 11 (2) ◽  
pp. 23
Author(s):  
Duy-Anh Nguyen ◽  
Xuan-Tu Tran ◽  
Francesca Iacopi
Keyword(s):  
Neural Networks ◽  
Computational Cost ◽  
Superior Performance ◽  
Training Algorithms ◽  
Current State ◽  
Hardware Implementations ◽  
Neuromorphic Hardware ◽  
High Level ◽  
Event Based ◽  
Hardware Platforms

Deep Learning (DL) has contributed to the success of many applications in recent years. The applications range from simple ones such as recognizing tiny images or simple speech patterns to ones with a high level of complexity such as playing the game of Go. However, this superior performance comes at a high computational cost, which made porting DL applications to conventional hardware platforms a challenging task. Many approaches have been investigated, and Spiking Neural Network (SNN) is one of the promising candidates. SNN is the third generation of Artificial Neural Networks (ANNs), where each neuron in the network uses discrete spikes to communicate in an event-based manner. SNNs have the potential advantage of achieving better energy efficiency than their ANN counterparts. While generally there will be a loss of accuracy on SNN models, new algorithms have helped to close the accuracy gap. For hardware implementations, SNNs have attracted much attention in the neuromorphic hardware research community. In this work, we review the basic background of SNNs, the current state and challenges of the training algorithms for SNNs and the current implementations of SNNs on various hardware platforms.

Optimizing the Learning Process of Feedforward Neural Networks Using Lightning Search Algorithm

2016 ◽  
Vol 25 (06) ◽  
pp. 1650033 ◽  
Author(s):  
Hossam Faris ◽  
Ibrahim Aljarah ◽  
Nailah Al-Madi ◽  
Seyedali Mirjalili
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Optimization Problems ◽  
Search Algorithm ◽  
Optimization Technique ◽  
Back Propagation ◽  
Feedforward Neural Networks ◽  
Training Algorithms ◽  
Local Optima ◽  
Local Solutions

Evolutionary Neural Networks are proven to be beneficial in solving challenging datasets mainly due to the high local optima avoidance. Stochastic operators in such techniques reduce the probability of stagnation in local solutions and assist them to supersede conventional training algorithms such as Back Propagation (BP) and Levenberg-Marquardt (LM). According to the No-Free-Lunch (NFL), however, there is no optimization technique for solving all optimization problems. This means that a Neural Network trained by a new algorithm has the potential to solve a new set of problems or outperform the current techniques in solving existing problems. This motivates our attempts to investigate the efficiency of the recently proposed Evolutionary Algorithm called Lightning Search Algorithm (LSA) in training Neural Network for the first time in the literature. The LSA-based trainer is benchmarked on 16 popular medical diagnosis problems and compared to BP, LM, and 6 other evolutionary trainers. The quantitative and qualitative results show that the LSA algorithm is able to show not only better local solutions avoidance but also faster convergence speed compared to the other algorithms employed. In addition, the statistical test conducted proves that the LSA-based trainer is significantly superior in comparison with the current algorithms on the majority of datasets.

Analyzing and Accelerating the Bottlenecks of Training Deep SNNs With Backpropagation

2020 ◽  
Vol 32 (12) ◽  
pp. 2557-2600
Author(s):  
Ruizhi Chen ◽  
Ling Li
Keyword(s):  
Neural Networks ◽  
Training Algorithms ◽  
Minimization Method ◽  
Data Set ◽  
Ultra Low Power ◽  
Training Performance ◽  
Speed Up ◽  
Pros And Cons ◽  
Accuracy Increase ◽  
Initialization Algorithm

Spiking neural networks (SNNs) with the event-driven manner of transmitting spikes consume ultra-low power on neuromorphic chips. However, training deep SNNs is still challenging compared to convolutional neural networks (CNNs). The SNN training algorithms have not achieved the same performance as CNNs. In this letter, we aim to understand the intrinsic limitations of SNN training to design better algorithms. First, the pros and cons of typical SNN training algorithms are analyzed. Then it is found that the spatiotemporal backpropagation algorithm (STBP) has potential in training deep SNNs due to its simplicity and fast convergence. Later, the main bottlenecks of the STBP algorithm are analyzed, and three conditions for training deep SNNs with the STBP algorithm are derived. By analyzing the connection between CNNs and SNNs, we propose a weight initialization algorithm to satisfy the three conditions. Moreover, we propose an error minimization method and a modified loss function to further improve the training performance. Experimental results show that the proposed method achieves 91.53% accuracy on the CIFAR10 data set with 1% accuracy increase over the STBP algorithm and decreases the training epochs on the MNIST data set to 15 epochs (over 13 times speed-up compared to the STBP algorithm). The proposed method also decreases classification latency by over 25 times compared to the CNN-SNN conversion algorithms. In addition, the proposed method works robustly for very deep SNNs, while the STBP algorithm fails in a 19-layer SNN.

scholarly journals Optical Recognition of Handwritten Logic Formulas Using Neural Networks

Electronics ◽  
2021 ◽  
Vol 10 (22) ◽  
pp. 2761
Author(s):  
Vaios Ampelakiotis ◽  
Isidoros Perikos ◽  
Ioannis Hatzilygeroudis ◽  
George Tsihrintzis
Keyword(s):  
Neural Networks ◽  
Character Recognition ◽  
Gradient Descent ◽  
Feedforward Neural Networks ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Training Algorithms ◽  
Gradient Descent Algorithm ◽  
Two Stages ◽  
And Training

In this paper, we present a handwritten character recognition (HCR) system that aims to recognize first-order logic handwritten formulas and create editable text files of the recognized formulas. Dense feedforward neural networks (NNs) are utilized, and their performance is examined under various training conditions and methods. More specifically, after three training algorithms (backpropagation, resilient propagation and stochastic gradient descent) had been tested, we created and trained an NN with the stochastic gradient descent algorithm, optimized by the Adam update rule, which was proved to be the best, using a trainset of 16,750 handwritten image samples of 28 × 28 each and a testset of 7947 samples. The final accuracy achieved is 90.13%. The general methodology followed consists of two stages: the image processing and the NN design and training. Finally, an application has been created that implements the methodology and automatically recognizes handwritten logic formulas. An interesting feature of the application is that it allows for creating new, user-oriented training sets and parameter settings, and thus new NN models.

Sign in / Sign up

Export Citation Format

Share Document