Neural Network Aided Information Theoretic Exploration

2018 ◽  
Author(s):  
Hantian Liu ◽  
M. Ani Hsieh
Keyword(s):  
Neural Network ◽  
Information Theoretic

PRUNING ARTIFICIAL NEURAL NETWORKS USING NEURAL COMPLEXITY MEASURES

2008 ◽  
Vol 18 (05) ◽  
pp. 389-403 ◽  
Author(s):  
THOMAS D. JORGENSEN ◽  
BARRY P. HAYNES ◽  
CHARLOTTE C. F. NORLUND
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Robot Control ◽  
Complexity Measures ◽  
Information Theoretic ◽  
Pruning Method ◽  
The Neural Network ◽  
Artificial Neural ◽  
Pruning Technique

This paper describes a new method for pruning artificial neural networks, using a measure of the neural complexity of the neural network. This measure is used to determine the connections that should be pruned. The measure computes the information-theoretic complexity of a neural network, which is similar to, yet different from previous research on pruning. The method proposed here shows how overly large and complex networks can be reduced in size, whilst retaining learnt behaviour and fitness. The technique proposed here helps to discover a network topology that matches the complexity of the problem it is meant to solve. This novel pruning technique is tested in a robot control domain, simulating a racecar. It is shown, that the proposed pruning method is a significant improvement over the most commonly used pruning method Magnitude Based Pruning. Furthermore, some of the pruned networks prove to be faster learners than the benchmark network that they originate from. This means that this pruning method can also help to unleash hidden potential in a network, because the learning time decreases substantially for a pruned a network, due to the reduction of dimensionality of the network.

Information theoretic subset selection for neural network models

1998 ◽  
Vol 22 (4-5) ◽  
pp. 613-626 ◽  
Author(s):  
Dasaratha V. Sridhar ◽  
Eric B. Bartlett ◽  
Richard C. Seagrave
Keyword(s):  
Neural Network ◽  
Subset Selection ◽  
Network Models ◽  
Neural Network Models ◽  
Information Theoretic ◽  
Selection For

scholarly journals Information theoretic bounds on cosmic string detection in CMB maps with noise

2019 ◽  
Vol 492 (1) ◽  
pp. 1329-1334 ◽  
Author(s):  
Razvan Ciuca ◽  
Oscar F Hernández
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Information Entropy ◽  
Posterior Distribution ◽  
Cosmic String ◽  
Noise Level ◽  
String Tension ◽  
Survey Area ◽  
Microwave Background ◽  
Information Theoretic

ABSTRACT We use a convolutional neural network to study cosmic string detection in cosmic microwave background (CMB) flat sky maps with Nambu–Goto strings. On noiseless maps, we can measure string tensions down to order 10−9, however when noise is included we are unable to measure string tensions below 10−7. Motivated by this impasse, we derive an information theoretic bound on the detection of the cosmic string tension Gμ from CMB maps. In particular, we bound the information entropy of the posterior distribution of Gμ in terms of the resolution, noise level and total survey area of the CMB map. We evaluate these bounds for the ACT, SPT-3G, Simons Observatory, Cosmic Origins Explorer, and CMB-S4 experiments. These bounds cannot be saturated by any method.

scholarly journals Topological Augmentation of Latent Information Streams in Feed-Forward Neural Networks

2020 ◽  
Author(s):  
James M. Shine ◽  
Mike Li ◽  
Oluwasanmi Koyejo ◽  
Ben Fulcher ◽  
Joseph T. Lizier
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Activity Patterns ◽  
Edge Weight ◽  
Feed Forward ◽  
Information Theoretic ◽  
Systems Neuroscience ◽  
Feed Forward Network ◽  
Low Dimensional

AbstractThe algorithmic rules that define deep neural networks are clearly defined, however the principles that define their performance remain poorly understood. Here, we use systems neuroscience and information theoretic approaches to analyse a feedforward neural network as it is trained to classify handwritten digits. By tracking the topology of the network as it learns, we identify three distinct phases of topological reconfiguration. Each phase brings the connections of the neural network into alignment with patterns of information contained in the input dataset, as well as the preceding layers. Performing dimensionality reduction on the data reveals a process of low-dimensional category separation as a function of learning. Our results enable a systems-level understanding of how deep neural networks function, and provide evidence of how neural networks reorganize edge weights and activity patterns so as to most effectively exploit the information theoretic content of input data during edge-weight training.SummaryTrained neural networks are capable of remarkable performance on complex categorization tasks, however the precise rules according to which the network reconfigures during training remain poorly understood. We used a combination of systems neuroscience and information theoretic analyses to interrogate the network topology of a simple, feed-forward network as it was trained on a digitclassification task. Over the course of training, the hidden layers of the network reconfigured in characteristic ways that were reminiscent of key results in network neuroscience studies of human brain imaging. In addition, we observed a strong correspondence between the topological changes at different learning phases and information theoretic signatures of the data that were entered into the network. In this way, we show how neural networks learn.

scholarly journals Determining the Rolling Window Size of Deep Neural Network Based Models on Time Series Forecasting

2021 ◽  
Vol 2078 (1) ◽  
pp. 012011
Author(s):  
Li Shen ◽  
Zijin Wei ◽  
Yangzhu Wang
Keyword(s):  
Neural Network ◽  
Time Series ◽  
Gaussian Noise ◽  
Deep Neural Network ◽  
Window Size ◽  
Time Series Forecasting ◽  
Time Series Models ◽  
Arima Models ◽  
Information Theoretic ◽  
Rolling Window

Abstract Time series forecasting has always been a significant task in various domains. In this paper, we propose DeepARMA, a LSTM-based recurrent neural network to tackle this problem. DeepARMA is derived from an existing time series forecasting baseline, DeepAR, overcoming two of its weaknesses: (1) rolling window size determination: the way DeepAR determines rolling window size is casual and vulnerable, which may lead to the unnecessary computation and inefficiency of the model;(2) neglect of the noise: pure autoregressive model cannot deal with the condition where data are composed of various kinds of noise, neither do most of time series models including DeepAR. In order to solve these two problems, we first combine a classic information theoretic criterion, AIC, with the network to determine the proper rolling window size. Then, we propose a jointly-learned neural network fusing white Gaussian noise series given by ARIMA models to DeepAR’s input. That is exactly why we name the network ‘DeepARMA’. Our experiments on a real-world dataset demonstrate that our improvement settles those two problems put forward above.

scholarly journals Information - Theoretic Methods for Anomaly Detection

2019 ◽  
pp. 21-29
Author(s):  
Mariam Haroutunian ◽  
Tigran Badasyan
Keyword(s):  
Neural Network ◽  
Anomaly Detection ◽  
Cyber Security ◽  
Digital Systems ◽  
Huge Amount ◽  
Information Theoretic ◽  
Normal Behavior ◽  
Network Methods ◽  
Pros And Cons ◽  
Information Theoretic Methods

Maintaining the security of digital systems with a huge amount of data is one of the main concerns of IT specialists in these times. Anomaly detection in systems is one of the solutions to overcome this challenge. Anomaly detection means ¯nding patterns that are not normal or deviate from normal behavior in a system. Anomaly detection has various applications in bio-informatics, image processing, cyber security, security for databases, etc. There are many groups of methods that are used for anomaly detection including statistical methods, neural network methods and information theoretic methods. In this paper we survey pros and cons of anomaly detection based on information theoretic techniques

Sign in / Sign up

Export Citation Format

Share Document