scholarly journals A Geometric Perspective on Information Plane Analysis

Entropy ◽  
2021 ◽  
Vol 23 (6) ◽  
pp. 711
Author(s):  
Mina Basirat ◽  
Bernhard C. Geiger ◽  
Peter M. Roth
Keyword(s):  
Neural Network ◽  
Mutual Information ◽  
Geometric Interpretation ◽  
Neural Network Training ◽  
Neural Network Learning ◽  
Network Learning ◽  
Plane Analysis ◽  
Network Training ◽  
Hidden Layer ◽  
The Impact

Information plane analysis, describing the mutual information between the input and a hidden layer and between a hidden layer and the target over time, has recently been proposed to analyze the training of neural networks. Since the activations of a hidden layer are typically continuous-valued, this mutual information cannot be computed analytically and must thus be estimated, resulting in apparently inconsistent or even contradicting results in the literature. The goal of this paper is to demonstrate how information plane analysis can still be a valuable tool for analyzing neural network training. To this end, we complement the prevailing binning estimator for mutual information with a geometric interpretation. With this geometric interpretation in mind, we evaluate the impact of regularization and interpret phenomena such as underfitting and overfitting. In addition, we investigate neural network learning in the presence of noisy data and noisy labels.

BINARY NEURAL NETWORK TRAINING ALGORITHMS BASED ON LINEAR SEQUENTIAL LEARNING

2003 ◽  
Vol 13 (05) ◽  
pp. 333-351 ◽  
Author(s):  
DI WANG ◽  
NARENDRA S. CHAUDHARI
Keyword(s):  
Neural Network ◽  
Sequential Learning ◽  
Training Algorithms ◽  
Neural Network Training ◽  
Neural Network Learning ◽  
Network Learning ◽  
Linear Separability ◽  
Network Training ◽  
Binary Neural Network ◽  
Hidden Layer

A key problem in Binary Neural Network learning is to decide bigger linear separable subsets. In this paper we prove some lemmas about linear separability. Based on these lemmas, we propose Multi-Core Learning (MCL) and Multi-Core Expand-and-Truncate Learning (MCETL) algorithms to construct Binary Neural Networks. We conclude that MCL and MCETL simplify the equations to compute weights and thresholds, and they result in the construction of simpler hidden layer. Examples are given to demonstrate these conclusions.

scholarly journals Evaluation of Parameter Settings for Training Neural Networks Using Backpropagation Algorithms

2020 ◽  
Vol 11 (4) ◽  
pp. 62-85
Author(s):  
Leema N. ◽  
Khanna H. Nehemiah ◽  
Elgin Christo V. R. ◽  
Kannan A.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Activation Function ◽  
Neural Network Training ◽  
Network Parameter ◽  
Network Parameters ◽  
Network Training ◽  
Hidden Layer ◽  
Function Number ◽  
The Impact

Artificial neural networks (ANN) are widely used for classification, and the training algorithm commonly used is the backpropagation (BP) algorithm. The major bottleneck faced in the backpropagation neural network training is in fixing the appropriate values for network parameters. The network parameters are initial weights, biases, activation function, number of hidden layers and the number of neurons per hidden layer, number of training epochs, learning rate, minimum error, and momentum term for the classification task. The objective of this work is to investigate the performance of 12 different BP algorithms with the impact of variations in network parameter values for the neural network training. The algorithms were evaluated with different training and testing samples taken from the three benchmark clinical datasets, namely, Pima Indian Diabetes (PID), Hepatitis, and Wisconsin Breast Cancer (WBC) dataset obtained from the University of California Irvine (UCI) machine learning repository.

Evaluation of Parameter Settings for Training Neural Networks Using Backpropagation Algorithms

2022 ◽  
pp. 202-226
Author(s):  
Leema N. ◽  
Khanna H. Nehemiah ◽  
Elgin Christo V. R. ◽  
Kannan A.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Activation Function ◽  
Neural Network Training ◽  
Network Parameter ◽  
Network Parameters ◽  
Network Training ◽  
Rate Minimum ◽  
Hidden Layer ◽  
Function Number

Artificial neural networks (ANN) are widely used for classification, and the training algorithm commonly used is the backpropagation (BP) algorithm. The major bottleneck faced in the backpropagation neural network training is in fixing the appropriate values for network parameters. The network parameters are initial weights, biases, activation function, number of hidden layers and the number of neurons per hidden layer, number of training epochs, learning rate, minimum error, and momentum term for the classification task. The objective of this work is to investigate the performance of 12 different BP algorithms with the impact of variations in network parameter values for the neural network training. The algorithms were evaluated with different training and testing samples taken from the three benchmark clinical datasets, namely, Pima Indian Diabetes (PID), Hepatitis, and Wisconsin Breast Cancer (WBC) dataset obtained from the University of California Irvine (UCI) machine learning repository.

scholarly journals The Application of BP Neural Network in Internet of Things

2012 ◽  
Vol 6-7 ◽  
pp. 1098-1102 ◽  
Author(s):  
Dan Dan Cui ◽  
Fei Liu
Keyword(s):  
Neural Network ◽  
Internet Of Things ◽  
Bp Neural Network ◽  
Learning Algorithm ◽  
Neural Network Learning ◽  
Network Learning ◽  
Input Layer ◽  
Telecommunications Network ◽  
Hidden Layer ◽  
The Internet Of Things

BP algorithm is a typical artificial neural network learning algorithm, the main structure consists of an input layer, one or more hidden layer, an output layer, the layers of the number of neurons, the output of each node the value is decided by the input values, the role, function and threshold. The Internet of Things is based on the information carrier of the traditional telecommunications network, so that all can be individually addressable ordinary physical objects to achieve the interoperability network. The paper puts forward the application of BP neural network in internet of things. The experiment shows BP is superior to RFID in internet of things.

scholarly journals Impact of Asymmetric Weight Update on Neural Network Training With Tiki-Taka Algorithm

2022 ◽  
Vol 15 ◽  
Author(s):  
Chaeun Lee ◽  
Kyungmi Noh ◽  
Wonjae Ji ◽  
Tayfun Gokmen ◽  
Seyoung Kim
Keyword(s):  
Neural Network ◽  
Power Efficiency ◽  
Calibration Method ◽  
Neural Network Training ◽  
Systematic Analysis ◽  
Training Performance ◽  
Actual Weight ◽  
Network Training ◽  
The Neural Network ◽  
The Impact

Recent progress in novel non-volatile memory-based synaptic device technologies and their feasibility for matrix-vector multiplication (MVM) has ignited active research on implementing analog neural network training accelerators with resistive crosspoint arrays. While significant performance boost as well as area- and power-efficiency is theoretically predicted, the realization of such analog accelerators is largely limited by non-ideal switching characteristics of crosspoint elements. One of the most performance-limiting non-idealities is the conductance update asymmetry which is known to distort the actual weight change values away from the calculation by error back-propagation and, therefore, significantly deteriorates the neural network training performance. To address this issue by an algorithmic remedy, Tiki-Taka algorithm was proposed and shown to be effective for neural network training with asymmetric devices. However, a systematic analysis to reveal the required asymmetry specification to guarantee the neural network performance has been unexplored. Here, we quantitatively analyze the impact of update asymmetry on the neural network training performance when trained with Tiki-Taka algorithm by exploring the space of asymmetry and hyper-parameters and measuring the classification accuracy. We discover that the update asymmetry level of the auxiliary array affects the way the optimizer takes the importance of previous gradients, whereas that of main array affects the frequency of accepting those gradients. We propose a novel calibration method to find the optimal operating point in terms of device and network parameters. By searching over the hyper-parameter space of Tiki-Taka algorithm using interpolation and Gaussian filtering, we find the optimal hyper-parameters efficiently and reveal the optimal range of asymmetry, namely the asymmetry specification. Finally, we show that the analysis and calibration method be applicable to spiking neural networks.

Fault Diagnosis Competitive Neural Network Training Through Condition Monitoring of Industrial Machines and Stock Exchange Prediction

2006 ◽  
Author(s):  
Sohrab Khanmohammadi ◽  
Sayyed Mahdi Hosseini
Keyword(s):  
Neural Network ◽  
Condition Monitoring ◽  
Stock Exchange ◽  
Analysis Data ◽  
Training Methods ◽  
Neural Network Training ◽  
Competitive Neural Network ◽  
Network Training ◽  
Highly Nonlinear ◽  
Hidden Layer

In this paper a new way for neural network training is introduced where the output of middle (hidden) layer of neural network is used to update weights in a competition procedure. Output layer’s weights are modified with multi layer perceptron (MLP) policy. This learning method is applied to two systems as case studies. First one is the monitoring of industrial machine where the results are compared with other training methods such as MLP or Radial Basis Function (RBF). Oil analysis data is used for condition monitoring. The data is gathered by using ten stages technique. The second one is the Stock prediction where the data are highly nonlinear and normally unpredictable especially when the markets are affected by political facts. The simulation results are analyzed and compared with other methods.

scholarly journals System Identification of Neural Signal Transmission Based on Backpropagation Neural Network

2020 ◽  
Vol 2020 ◽  
pp. 1-8
Author(s):  
Xiangyu Li ◽  
Chunhua Yuan ◽  
Bonan Shan
Keyword(s):  
Neural Network ◽  
Bp Neural Network ◽  
Signal Transmission ◽  
Neural Model ◽  
Neural Firing ◽  
Output Data ◽  
Neural Network Learning ◽  
Network Learning ◽  
Network Training ◽  
Input And Output

The identification method of backpropagation (BP) neural network is adopted to approximate the mapping relation between input and output of neurons based on neural firing trajectory in this paper. In advance, the input and output data of neural model is used for BP neural network learning, so that the identified BP neural network can present the transfer characteristics of the model, which makes the network precisely predict the firing trajectory of the neural model. In addition, the method is applied to identify electrophysiological experimental data of real neurons, so that the output of the identified BP neural network can not only accurately fit the neural firing trajectories of neurons participating in the network training but also predict the firing trajectories and spike moments of neurons which are not involved in the training process with high accuracy.

Non-linear time series analysis of combustion pressure data for neural network training with the concept of mutual information

2001 ◽  
Vol 215 (2) ◽  
pp. 299-304 ◽  
Author(s):  
F Heister ◽  
M Froehlich
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Mutual Information ◽  
Combustion Engine ◽  
Linear Time ◽  
Chamber Pressure ◽  
Neural Network Training ◽  
Input Output ◽  
Output Data ◽  
Network Training

In recent years, after a period of disillusion in the field of neural processing and adaptive algorithms, neural networks have been reconsidered for solving complex technical tasks. The problem of neural network training is the presentation of input/output data showing an appropriate information content which represent a given problem. The training of a neural structure will definitely lead to poor results if the relation between input and output signals shows no functional dependence but a pure stochastic behaviour. This paper is concerned with the identification of the most relevant input-output data pairs for neural networks, using the concept of mutual information. A general, quantitative method is demonstrated for identifying the most relevant points from the transient measured data of a combustion engine. In this context mutual information is employed for the problem of determining the 50 per cent energy conversion point solely from the combustion chamber pressure during one combustion cycle.

scholarly journals OPTIMIZATION OF THE INPUT DATA VECTOR TO IMPROVE THE NEURAL NETWORK TRAINING FOR OPC

2021 ◽  
Author(s):  
Georgiy Teplov ◽  
Almira Galeeva ◽  
Aleksey Kuzovkov
Keyword(s):  
Neural Network ◽  
Data Structure ◽  
Input Data ◽  
Input Vector ◽  
Neural Network Training ◽  
Network Training ◽  
The Neural Network ◽  
Data Vector ◽  
The Impact

This work explored the impact of input data structure to improve the neural network training. The impact of two variants of the input data vector on the training accuracy of the network was studied. The first version of the input vector included the intensity of the exposure radiation map. The second version of the input vector included the intensity of the exposure radiation map and IC topology.

Sign in / Sign up

Export Citation Format

Share Document