Recursive Construction of Stable Assemblies of Recurrent Neural Networks
- URL: http://arxiv.org/abs/2106.08928v1
- Date: Wed, 16 Jun 2021 16:35:50 GMT
- Title: Recursive Construction of Stable Assemblies of Recurrent Neural Networks
- Authors: Michaela Ennis, Leo Kozachkov, Jean-Jacques Slotine
- Abstract summary: Advanced applications of machine learning will likely involve combinations of trained networks.
This paper takes a step in this direction by establishing contraction properties of broad classes of nonlinear recurrent networks and neural ODEs.
Results can be used to combine recurrent networks and physical systems with quantified contraction properties.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Advanced applications of modern machine learning will likely involve
combinations of trained networks, as are already used in spectacular systems
such as DeepMind's AlphaGo. Recursively building such combinations in an
effective and stable fashion while also allowing for continual refinement of
the individual networks - as nature does for biological networks - will require
new analysis tools. This paper takes a step in this direction by establishing
contraction properties of broad classes of nonlinear recurrent networks and
neural ODEs, and showing how these quantified properties allow in turn to
recursively construct stable networks of networks in a systematic fashion. The
results can also be used to stably combine recurrent networks and physical
systems with quantified contraction properties. Similarly, they may be applied
to modular computational models of cognition.
Related papers
- Coding schemes in neural networks learning classification tasks [52.22978725954347]
We investigate fully-connected, wide neural networks learning classification tasks.
We show that the networks acquire strong, data-dependent features.
Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
arXiv Detail & Related papers (2024-06-24T14:50:05Z) - Symbolic Regression of Dynamic Network Models [0.0]
We introduce a novel formulation of a network generator and a parameter-free fitness function to evaluate the generated network.
We extend this approach by modifying generator semantics to create and retrieve rules for time-varying networks.
The framework was then used on three empirical datasets - subway networks of major cities, regions of street networks and semantic co-occurrence networks of literature in Artificial Intelligence.
arXiv Detail & Related papers (2023-12-15T00:34:45Z) - Riemannian Residual Neural Networks [58.925132597945634]
We show how to extend the residual neural network (ResNet)
ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks.
arXiv Detail & Related papers (2023-10-16T02:12:32Z) - How neural networks learn to classify chaotic time series [77.34726150561087]
We study the inner workings of neural networks trained to classify regular-versus-chaotic time series.
We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
arXiv Detail & Related papers (2023-06-04T08:53:27Z) - Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems [9.382423715831687]
In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems.
Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks.
arXiv Detail & Related papers (2022-09-22T10:32:08Z) - Dimensionality Reduction in Deep Learning via Kronecker Multi-layer
Architectures [4.836352379142503]
We propose a new deep learning architecture based on fast matrix multiplication of a Kronecker product decomposition.
We show that this architecture allows a neural network to be trained and implemented with a significant reduction in computational time and resources.
arXiv Detail & Related papers (2022-04-08T19:54:52Z) - Quasi-orthogonality and intrinsic dimensions as measures of learning and
generalisation [55.80128181112308]
We show that dimensionality and quasi-orthogonality of neural networks' feature space may jointly serve as network's performance discriminants.
Our findings suggest important relationships between the networks' final performance and properties of their randomly initialised feature spaces.
arXiv Detail & Related papers (2022-03-30T21:47:32Z) - Creating Powerful and Interpretable Models withRegression Networks [2.2049183478692584]
We propose a novel architecture, Regression Networks, which combines the power of neural networks with the understandability of regression analysis.
We demonstrate that the models exceed the state-of-the-art performance of interpretable models on several benchmark datasets.
arXiv Detail & Related papers (2021-07-30T03:37:00Z) - Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
We propose a topological perspective to represent a network into a complete graph for analysis.
By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner.
This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
arXiv Detail & Related papers (2020-08-19T04:53:31Z) - Neural networks adapting to datasets: learning network size and topology [77.34726150561087]
We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a gradient-based training.
The resulting network has the structure of a graph tailored to the particular learning task and dataset.
arXiv Detail & Related papers (2020-06-22T12:46:44Z) - Input-to-State Representation in linear reservoirs dynamics [15.491286626948881]
Reservoir computing is a popular approach to design recurrent neural networks.
The working principle of these networks is not fully understood.
A novel analysis of the dynamics of such networks is proposed.
arXiv Detail & Related papers (2020-03-24T00:14:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.