Understanding State Space Organization in Recurrent Neural Networks with Iterative Function Systems Dynamics

2000 ◽  
pp. 255-269 ◽  
Author(s):  
Peter Tiňo ◽  
Georg Dorffner ◽  
Christian Schittenkopf
Keyword(s):  
Neural Networks ◽  
State Space ◽  
Recurrent Neural Networks ◽  
Systems Dynamics ◽  
Space Organization

scholarly journals Reservoir computing and the Sooner-is-Better bottleneck

2016 ◽  
Vol 39 ◽  
Author(s):  
Stefan L. Frank ◽  
Hartmut Fitz
Keyword(s):  
Neural Networks ◽  
State Space ◽  
Recurrent Neural Networks ◽  
Random Point ◽  
High Dimensional ◽  
Reservoir Computing ◽  
Current Input ◽  
Dimensional State Space ◽  
Language Input

AbstractPrior language input is not lost but integrated with the current input. This principle is demonstrated by “reservoir computing”: Untrained recurrent neural networks project input sequences onto a random point in high-dimensional state space. Earlier inputs can be retrieved from this projection, albeit less reliably so as more input is received. The bottleneck is therefore not “Now-or-Never” but “Sooner-is-Better.”

scholarly journals Differential geometry methods for constructing manifold-targeted recurrent neural networks.

2021 ◽  
Author(s):  
Federico Claudi ◽  
Tiago Branco
Keyword(s):  
Neural Networks ◽  
Differential Geometry ◽  
State Space ◽  
Recurrent Neural Networks ◽  
Network Connectivity ◽  
Neural Dynamics ◽  
Target Manifold ◽  
Neural Computations ◽  
Dimensional State Space ◽  
Low Dimensional

Neural computations can be framed as dynamical processes, whereby the structure of the dynamics within a neural network are a direct reflection of the computations that the network performs. A key step in generating mechanistic interpretations within this computation through dynamics framework is to establish the link between network connectivity, dynamics and computation. This link is only partly understood. Recent work has focused on producing algorithms for engineering artificial recurrent neural networks (RNN) with dynamics targeted to a specific goal manifold. Some of these algorithms only require a set of vectors tangent to the target manifold to be computed, and thus provide a general method that can be applied to a diverse set of problems. Nevertheless, computing such vectors for an arbitrary manifold in a high dimensional state space remains highly challenging, which in practice limits the applicability of this approach. Here we demonstrate how topology and differential geometry can be leveraged to simplify this task, by first computing tangent vectors on a low-dimensional topological manifold and then embedding these in state space. The simplicity of this procedure greatly facilitates the creation of manifold-targeted RNNs, as well as the process of designing task-solving on-manifold dynamics. This new method should enable the application of network engineering-based approaches to a wide set of problems in neuroscience and machine learning. Furthermore, our description of how fundamental concepts from differential geometry can be mapped onto different aspects of neural dynamics is a further demonstration of how the language of differential geometry can enrich the conceptual framework for describing neural dynamics and computation.

Sign in / Sign up

Export Citation Format

Share Document