Enhanced Gradient Descent Algorithms for Complex-Valued Neural Networks

2014 ◽  
Author(s):  
Calin-Adrian Popa
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Descent Algorithms ◽  
Complex Valued

scholarly journals Gradient-Sensitive Optimization for Convolutional Neural Networks

2021 ◽  
Vol 2021 ◽  
pp. 1-16
Author(s):  
Zhipeng Liu ◽  
Rui Feng ◽  
Xiuhan Li ◽  
Wei Wang ◽  
Xiaoling Wu
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Adaptive Learning ◽  
Gradient Descent ◽  
Recognition Rate ◽  
Learning Rate ◽  
Basic Algorithm ◽  
Descent Algorithms ◽  
Moment Estimation ◽  
Effective Models

Convolutional neural networks (CNNs) are effective models for image classification and recognition. Gradient descent optimization (GD) is the basic algorithm for CNN model optimization. Since GD appeared, a series of improved algorithms have been derived. Among these algorithms, adaptive moment estimation (Adam) has been widely recognized. However, local changes are ignored in Adam to some extent. In this paper, we introduce an adaptive learning rate factor based on current and recent gradients. According to this factor, we can dynamically adjust the learning rate of each independent parameter to adaptively adjust the global convergence process. We use the factor to adjust the learning rate for each parameter. The convergence of the proposed algorithm is proven by using the regret bound approach of the online learning framework. In the experimental section, comparisons are conducted between the proposed algorithm and other existing algorithms, such as AdaGrad, RMSprop, Adam, diffGrad, and AdaHMG, on test functions and the MNIST dataset. The results show that Adam and RMSprop combined with our algorithm can not only find the global minimum faster in the experiment using the test function but also have a better convergence curve and higher test set accuracy in experiments using datasets. Our algorithm is a supplement to the existing gradient descent algorithms, which can be combined with many other existing gradient descent algorithms to improve the efficiency of iteration, speed up the convergence of the cost function, and improve the final recognition rate.

PSEUDO-RELAXATION LEARNING ALGORITHM FOR COMPLEX-VALUED ASSOCIATIVE MEMORY

2008 ◽  
Vol 18 (02) ◽  
pp. 147-156 ◽  
Author(s):  
MASAKI KOBAYASHI
Keyword(s):  
Neural Networks ◽  
Associative Memory ◽  
Gradient Descent ◽  
Storage Capacity ◽  
Learning Algorithm ◽  
Inverse Matrix ◽  
Associative Memories ◽  
Hebb Rule ◽  
Complex Valued ◽  
Pseudo Inverse

HAM (Hopfield Associative Memory) and BAM (Bidirectinal Associative Memory) are representative associative memories by neural networks. The storage capacity by the Hebb rule, which is often used, is extremely low. In order to improve it, some learning methods, for example, pseudo-inverse matrix learning and gradient descent learning, have been introduced. Oh introduced pseudo-relaxation learning algorithm to HAM and BAM. In order to accelerate it, Hattori proposed quick learning. Noest proposed CAM (Complex-valued Associative Memory), which is complex-valued HAM. The storage capacity of CAM by the Hebb rule is also extremely low. Pseudo-inverse matrix learning and gradient descent learning have already been generalized to CAM. In this paper, we apply pseudo-relaxation learning algorithm to CAM in order to improve the capacity.

Sign in / Sign up

Export Citation Format

Share Document