scholarly journals Convergence of an Online Split-Complex Gradient Algorithm for Complex-Valued Neural Networks

2010 ◽  
Vol 2010 ◽  
pp. 1-27
Author(s):  
Huisheng Zhang ◽  
Dongpo Xu ◽  
Zhiping Wang
Keyword(s):  
Neural Networks ◽  
Fixed Point ◽  
Adaptive Learning ◽  
Gradient Method ◽  
Error Function ◽  
Gradient Algorithm ◽  
Training Procedure ◽  
Weight Sequence ◽  
Complex Valued ◽  
Complex Gradient

The online gradient method has been widely used in training neural networks. We consider in this paper an online split-complex gradient algorithm for complex-valued neural networks. We choose an adaptive learning rate during the training procedure. Under certain conditions, by firstly showing the monotonicity of the error function, it is proved that the gradient of the error function tends to zero and the weight sequence tends to a fixed point. A numerical example is given to support the theoretical findings.

Convergence Analysis of Three Classes of Split-Complex Gradient Algorithms for Complex-Valued Recurrent Neural Networks

2010 ◽  
Vol 22 (10) ◽  
pp. 2655-2677 ◽  
Author(s):  
Dongpo Xu ◽  
Huisheng Zhang ◽  
Lijun Liu
Keyword(s):  
Neural Networks ◽  
Convergence Analysis ◽  
Recurrent Neural Networks ◽  
Error Function ◽  
Iteration Process ◽  
Convergence Result ◽  
Stationary Points ◽  
Gradient Algorithms ◽  
Complex Valued ◽  
Complex Gradient

This letter presents a unified convergence analysis of the split-complex nonlinear gradient descent (SCNGD) learning algorithms for complex-valued recurrent neural networks, covering three classes of SCNGD algorithms: standard SCNGD, normalized SCNGD, and adaptive normalized SCNGD. We prove that if the activation functions are of split-complex type and some conditions are satisfied, the error function is monotonically decreasing during the training iteration process, and the gradients of the error function with respect to the real and imaginary parts of the weights converge to zero. A strong convergence result is also obtained under the assumption that the error function has only a finite number of stationary points. The simulation results are given to support the theoretical analysis.

Training Pi-Sigma Network by Online Gradient Algorithm with Penalty for Small Weight Update

2007 ◽  
Vol 19 (12) ◽  
pp. 3356-3368 ◽  
Author(s):  
Yan Xiong ◽  
Wei Wu ◽  
Xidai Kang ◽  
Chao Zhang
Keyword(s):  
Neural Networks ◽  
Numerical Experiments ◽  
Error Function ◽  
Feedforward Neural Networks ◽  
Gradient Algorithm ◽  
Training Method ◽  
Penalty Term ◽  
Slow Convergence ◽  
Output Layer ◽  
Adaptive Penalty

A pi-sigma network is a class of feedforward neural networks with product units in the output layer. An online gradient algorithm is the simplest and most often used training method for feedforward neural networks. But there arises a problem when the online gradient algorithm is used for pi-sigma networks in that the update increment of the weights may become very small, especially early in training, resulting in a very slow convergence. To overcome this difficulty, we introduce an adaptive penalty term into the error function, so as to increase the magnitude of the update increment of the weights when it is too small. This strategy brings about faster convergence as shown by the numerical experiments carried out in this letter.

scholarly journals Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks

2009 ◽  
Vol 2009 ◽  
pp. 1-16 ◽  
Author(s):  
Huisheng Zhang ◽  
Chao Zhang ◽  
Wei Wu
Keyword(s):  
Neural Networks ◽  
Theoretical Analysis ◽  
Error Function ◽  
Iteration Process ◽  
Learning Rate ◽  
Backpropagation Algorithm ◽  
Numerical Example ◽  
Moderate Condition ◽  
Complex Valued

The batch split-complex backpropagation (BSCBP) algorithm for training complex-valued neural networks is considered. For constant learning rate, it is proved that the error function of BSCBP algorithm is monotone during the training iteration process, and the gradient of the error function tends to zero. By adding a moderate condition, the weights sequence itself is also proved to be convergent. A numerical example is given to support the theoretical analysis.

A MODIFIED ERROR BACKPROPAGATION ALGORITHM FOR COMPLEX-VALUE NEURAL NETWORKS

2005 ◽  
Vol 15 (06) ◽  
pp. 435-443 ◽  
Author(s):  
XIAOMING CHEN ◽  
ZHENG TANG ◽  
CATHERINE VARIAPPAN ◽  
SONGSONG LI ◽  
TOSHIMI OKADA
Keyword(s):  
Neural Networks ◽  
Image Processing ◽  
Fourier Transformation ◽  
Error Function ◽  
Local Minima ◽  
Backpropagation Algorithm ◽  
Speed Up ◽  
Simulation Results ◽  
Hidden Layer ◽  
Complex Valued

The complex-valued backpropagation algorithm has been widely used in fields of dealing with telecommunications, speech recognition and image processing with Fourier transformation. However, the local minima problem usually occurs in the process of learning. To solve this problem and to speed up the learning process, we propose a modified error function by adding a term to the conventional error function, which is corresponding to the hidden layer error. The simulation results show that the proposed algorithm is capable of preventing the learning from sticking into the local minima and of speeding up the learning.

Sign in / Sign up

Export Citation Format

Share Document