scholarly journals Lessons From Deep Neural Networks for Studying the Coding Principles of Biological Neural Networks

2021 ◽  
Vol 14 ◽  
Author(s):  
Hyojin Bae ◽  
Sang Jeong Kim ◽  
Chang-Eop Kim
Keyword(s):  
Neural Networks ◽  
Neural Coding ◽  
Deep Neural Networks ◽  
Neural Response ◽  
Feature Representation ◽  
Network Feature ◽  
Biological Neural Networks ◽  
Stimulus Features ◽  
The Comparative Study ◽  
The Brain

One of the central goals in systems neuroscience is to understand how information is encoded in the brain, and the standard approach is to identify the relation between a stimulus and a neural response. However, the feature of a stimulus is typically defined by the researcher's hypothesis, which may cause biases in the research conclusion. To demonstrate potential biases, we simulate four likely scenarios using deep neural networks trained on the image classification dataset CIFAR-10 and demonstrate the possibility of selecting suboptimal/irrelevant features or overestimating the network feature representation/noise correlation. Additionally, we present studies investigating neural coding principles in biological neural networks to which our points can be applied. This study aims to not only highlight the importance of careful assumptions and interpretations regarding the neural response to stimulus features but also suggest that the comparative study between deep and biological neural networks from the perspective of machine learning can be an effective strategy for understanding the coding principles of the brain.

scholarly journals Brain hierarchy score: Which deep neural networks are hierarchically brain-like?

2020 ◽  
Author(s):  
Soma Nonaka ◽  
Kei Majima ◽  
Shuntaro C. Aoki ◽  
Yukiyasu Kamitani
Keyword(s):  
Neural Networks ◽  
Image Recognition ◽  
High Performance ◽  
Deep Neural Networks ◽  
Recognition Performance ◽  
Brain Activity ◽  
Spatial Integration ◽  
Hierarchical Processing ◽  
Visual Areas ◽  
The Brain

SummaryAchievement of human-level image recognition by deep neural networks (DNNs) has spurred interest in whether and how DNNs are brain-like. Both DNNs and the visual cortex perform hierarchical processing, and correspondence has been shown between hierarchical visual areas and DNN layers in representing visual features. Here, we propose the brain hierarchy (BH) score as a metric to quantify the degree of hierarchical correspondence based on the decoding of individual DNN unit activations from human brain activity. We find that BH scores for 29 pretrained DNNs with varying architectures are negatively correlated with image recognition performance, indicating that recently developed high-performance DNNs are not necessarily brain-like. Experimental manipulations of DNN models suggest that relatively simple feedforward architecture with broad spatial integration is critical to brain-like hierarchy. Our method provides new ways for designing DNNs and understanding the brain in consideration of their representational homology.

scholarly journals Attention Promotes the Neural Encoding of Prediction Errors

2019 ◽  
Author(s):  
Cooper A. Smout ◽  
Matthew F. Tang ◽  
Marta I. Garrido ◽  
Jason B. Mattingley
Keyword(s):  
Information Processing ◽  
Coding Theory ◽  
Predictive Coding ◽  
Sensory Information ◽  
Neural Response ◽  
Prediction Errors ◽  
Feature Information ◽  
The Difference ◽  
Stimulus Features ◽  
The Brain

AbstractThe human brain is thought to optimise the encoding of incoming sensory information through two principal mechanisms: prediction uses stored information to guide the interpretation of forthcoming sensory events, and attention prioritizes these events according to their behavioural relevance. Despite the ubiquitous contributions of attention and prediction to various aspects of perception and cognition, it remains unknown how they interact to modulate information processing in the brain. A recent extension of predictive coding theory suggests that attention optimises the expected precision of predictions by modulating the synaptic gain of prediction error units. Since prediction errors code for the difference between predictions and sensory signals, this model would suggest that attention increases the selectivity for mismatch information in the neural response to a surprising stimulus. Alternative predictive coding models proposes that attention increases the activity of prediction (or ‘representation’) neurons, and would therefore suggest that attention and prediction synergistically modulate selectivity for feature information in the brain. Here we applied multivariate forward encoding techniques to neural activity recorded via electroencephalography (EEG) as human observers performed a simple visual task, to test for the effect of attention on both mismatch and feature information in the neural response to surprising stimuli. Participants attended or ignored a periodic stream of gratings, the orientations of which could be either predictable, surprising, or unpredictable. We found that surprising stimuli evoked neural responses that were encoded according to the difference between predicted and observed stimulus features, and that attention facilitated the encoding of this type of information in the brain. These findings advance our understanding of how attention and prediction modulate information processing in the brain, and support the theory that attention optimises precision expectations during hierarchical inference by increasing the gain of prediction errors.

scholarly journals Understanding a Deep Learning Technique through a Neuromorphic System a Case Study with SpiNNaker Neuromorphic Platform

2018 ◽  
Vol 164 ◽  
pp. 01015
Author(s):  
Indar Sugiarto ◽  
Felix Pasila
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Cellular Level ◽  
System A ◽  
Learning Technique ◽  
Neuromorphic System ◽  
The Brain ◽  
Potential Use

Deep learning (DL) has been considered as a breakthrough technique in the field of artificial intelligence and machine learning. Conceptually, it relies on a many-layer network that exhibits a hierarchically non-linear processing capability. Some DL architectures such as deep neural networks, deep belief networks and recurrent neural networks have been developed and applied to many fields with incredible results, even comparable to human intelligence. However, many researchers are still sceptical about its true capability: can the intelligence demonstrated by deep learning technique be applied for general tasks? This question motivates the emergence of another research discipline: neuromorphic computing (NC). In NC, researchers try to identify the most fundamental ingredients that construct intelligence behaviour produced by the brain itself. To achieve this, neuromorphic systems are developed to mimic the brain functionality down to cellular level. In this paper, a neuromorphic platform called SpiNNaker is described and evaluated in order to understand its potential use as a platform for a deep learning approach. This paper is a literature review that contains comparative study on algorithms that have been implemented in SpiNNaker.

scholarly journals Biomedical event extraction with a novel combination strategy based on hybrid deep neural networks

2020 ◽  
Vol 21 (1) ◽  
Author(s):  
Lvxing Zhu ◽  
Haoran Zheng
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Feature Representation ◽  
Event Extraction ◽  
Combination Strategy ◽  
Score Improvement ◽  
Biomedical Event Extraction ◽  
Complex Events ◽  
Substantial Interest

Abstract Background Biomedical event extraction is a fundamental and in-demand technology that has attracted substantial interest from many researchers. Previous works have heavily relied on manual designed features and external NLP packages in which the feature engineering is large and complex. Additionally, most of the existing works use the pipeline process that breaks down a task into simple sub-tasks but ignores the interaction between them. To overcome these limitations, we propose a novel event combination strategy based on hybrid deep neural networks to settle the task in a joint end-to-end manner. Results We adapted our method to several annotated corpora of biomedical event extraction tasks. Our method achieved state-of-the-art performance with noticeable overall F1 score improvement compared to that of existing methods for all of these corpora. Conclusions The experimental results demonstrated that our method is effective for biomedical event extraction. The combination strategy can reconstruct complex events from the output of deep neural networks, while the deep neural networks effectively capture the feature representation from the raw text. The biomedical event extraction implementation is available online at http://www.predictor.xin/event_extraction.

scholarly journals A Correspondence between Normalization Strategies in Artificial and Biological Neural Networks

2021 ◽  
pp. 1-25
Author(s):  
Yang Shen ◽  
Julia Wang ◽  
Saket Navlakha
Keyword(s):  
Neural Networks ◽  
Spatial Scales ◽  
Homeostatic Plasticity ◽  
Network Activity ◽  
Data Sets ◽  
Normal Brain ◽  
Trade Offs ◽  
Normalization Methods ◽  
Biological Neural Networks ◽  
The Brain

Abstract A fundamental challenge at the interface of machine learning and neuroscience is to uncover computational principles that are shared between artificial and biological neural networks. In deep learning, normalization methods such as batch normalization, weight normalization, and their many variants help to stabilize hidden unit activity and accelerate network training, and these methods have been called one of the most important recent innovations for optimizing deep networks. In the brain, homeostatic plasticity represents a set of mechanisms that also stabilize and normalize network activity to lie within certain ranges, and these mechanisms are critical for maintaining normal brain function. In this article, we discuss parallels between artificial and biological normalization methods at four spatial scales: normalization of a single neuron's activity, normalization of synaptic weights of a neuron, normalization of a layer of neurons, and normalization of a network of neurons. We argue that both types of methods are functionally equivalent—that is, both push activation patterns of hidden units toward a homeostatic state, where all neurons are equally used—and we argue that such representations can improve coding capacity, discrimination, and regularization. As a proof of concept, we develop an algorithm, inspired by a neural normalization technique called synaptic scaling, and show that this algorithm performs competitively against existing normalization methods on several data sets. Overall, we hope this bidirectional connection will inspire neuroscientists and machine learners in three ways: to uncover new normalization algorithms based on established neurobiological principles; to help quantify the trade-offs of different homeostatic plasticity mechanisms used in the brain; and to offer insights about how stability may not hinder, but may actually promote, plasticity.

scholarly journals RCA: A Deep Collaborative Autoencoder Approach for Anomaly Detection

2021 ◽  
Author(s):  
Boyang Liu ◽  
Ding Wang ◽  
Kaixiang Lin ◽  
Pang-Ning Tan ◽  
Jiayu Zhou
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Anomaly Detection ◽  
Missing Values ◽  
Deep Neural Networks ◽  
Reconstruction Error ◽  
Feature Representation ◽  
High Complexity ◽  
Unsupervised Anomaly Detection ◽  
Robust Framework

Unsupervised anomaly detection plays a crucial role in many critical applications. Driven by the success of deep learning, recent years have witnessed growing interests in applying deep neural networks (DNNs) to anomaly detection problems. A common approach is using autoencoders to learn a feature representation for the normal observations in the data. The reconstruction error of the autoencoder is then used as outlier scores to detect the anomalies. However, due to the high complexity brought upon by the over-parameterization of DNNs, the reconstruction error of the anomalies could also be small, which hampers the effectiveness of these methods. To alleviate this problem, we propose a robust framework using collaborative autoencoders to jointly identify normal observations from the data while learning its feature representation. We investigate the theoretical properties of the framework and empirically show its outstanding performance as compared to other DNN-based methods. Our experimental results also show the resiliency of the framework to missing values compared to other baseline methods.

The architecture challenge: Future artificial-intelligence systems will require sophisticated architectures, and knowledge of the brain might guide their construction

2017 ◽  
Vol 40 ◽  
Author(s):  
Gianluca Baldassarre ◽  
Vieri Giuliano Santucci ◽  
Emilio Cartoni ◽  
Daniele Caligiore
Keyword(s):  
Artificial Intelligence ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Human Cognition ◽  
Important Means ◽  
The Brain ◽  
Artificial Intelligence Systems

AbstractIn this commentary, we highlight a crucial challenge posed by the proposal of Lake et al. to introduce key elements of human cognition into deep neural networks and future artificial-intelligence systems: the need to design effective sophisticated architectures. We propose that looking at the brain is an important means of facing this great challenge.

scholarly journals Brain Age Prediction With Morphological Features Using Deep Neural Networks: Results From Predictive Analytic Competition 2019

2021 ◽  
Vol 11 ◽  
Author(s):  
Angela Lombardi ◽  
Alfonso Monaco ◽  
Giacinto Donvito ◽  
Nicola Amoroso ◽  
Roberto Bellotti ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Morphological Changes ◽  
Structure Characterization ◽  
Feature Importance ◽  
Mri Scans ◽  
Magnetic Resonance Imaging Mri ◽  
Brain Age ◽  
The Brain ◽  
Age Prediction

Morphological changes in the brain over the lifespan have been successfully described by using structural magnetic resonance imaging (MRI) in conjunction with machine learning (ML) algorithms. International challenges and scientific initiatives to share open access imaging datasets also contributed significantly to the advance in brain structure characterization and brain age prediction methods. In this work, we present the results of the predictive model based on deep neural networks (DNN) proposed during the Predictive Analytic Competition 2019 for brain age prediction of 2638 healthy individuals. We used FreeSurfer software to extract some morphological descriptors from the raw MRI scans of the subjects collected from 17 sites. We compared the proposed DNN architecture with other ML algorithms commonly used in the literature (RF, SVR, Lasso). Our results highlight that the DNN models achieved the best performance with MAE = 4.6 on the hold-out test, outperforming the other ML strategies. We also propose a complete ML framework to perform a robust statistical evaluation of feature importance for the clinical interpretability of the results.

scholarly journals Do deep neural networks see the way we do?

2019 ◽  
Author(s):  
Georgin Jacob ◽  
R. T. Pramod ◽  
Harish Katti ◽  
S. P. Arun
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Relative Size ◽  
Training Data ◽  
Object Representations ◽  
Multiple Dimensions ◽  
Deep Networks ◽  
The Brain ◽  
Global Advantage ◽  
The Way

ABSTRACTDeep neural networks have revolutionized computer vision, and their object representations match coarsely with the brain. As a result, it is widely believed that any fine scale differences between deep networks and brains can be fixed with increased training data or minor changes in architecture. But what if there are qualitative differences between brains and deep networks? Do deep networks even see the way we do? To answer this question, we chose a deep neural network optimized for object recognition and asked whether it exhibits well-known perceptual and neural phenomena despite not being explicitly trained to do so. To our surprise, many phenomena were present in the network, including the Thatcher effect, mirror confusion, Weber’s law, relative size, multiple object normalization and sparse coding along multiple dimensions. However, some perceptual phenomena were notably absent, including processing of 3D shape, patterns on surfaces, occlusion, natural parts and a global advantage. Our results elucidate the computational challenges of vision by showing that learning to recognize objects suffices to produce some perceptual phenomena but not others and reveal the perceptual properties that could be incorporated into deep networks to improve their performance.

Sign in / Sign up

Export Citation Format

Share Document