Approximation Bounds for Random Neural Networks and Reservoir Systems

• URL: http://arxiv.org/abs/2002.05933v2
• Date: Tue, 16 Feb 2021 09:14:59 GMT
• Title: Approximation Bounds for Random Neural Networks and Reservoir Systems
• Authors: Lukas Gonon, Lyudmila Grigoryeva, and Juan-Pablo Ortega
• Abstract summary: This work studies approximation based on single-hidden-layer feedforward and recurrent neural networks with randomly generated internal weights. In particular, this proves that echo state networks with randomly generated weights are capable of approximating a wide class of dynamical systems arbitrarily well.
• Score: 8.143750358586072
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work studies approximation based on single-hidden-layer feedforward and recurrent neural networks with randomly generated internal weights. These methods, in which only the last layer of weights and a few hyperparameters are optimized, have been successfully applied in a wide range of static and dynamic learning problems. Despite the popularity of this approach in empirical tasks, important theoretical questions regarding the relation between the unknown function, the weight distribution, and the approximation rate have remained open. In this work it is proved that, as long as the unknown function, functional, or dynamical system is sufficiently regular, it is possible to draw the internal weights of the random (recurrent) neural network from a generic distribution (not depending on the unknown object) and quantify the error in terms of the number of neurons and the hyperparameters. In particular, this proves that echo state networks with randomly generated weights are capable of approximating a wide class of dynamical systems arbitrarily well and thus provides the first mathematical explanation for their empirically observed success at learning dynamical systems.

Related papers

• Expressivity of Neural Networks with Random Weights and Learned Biases [44.02417750529102]
  Recent work has pushed the bounds of universal approximation by showing that arbitrary functions can similarly be learned by tuning smaller subsets of parameters. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can be trained to perform multiple tasks by learning biases only. Our results are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights.
  arXiv  Detail & Related papers  (2024-07-01T04:25:49Z)
• Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
  We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence. We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers. This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
  arXiv  Detail & Related papers  (2023-07-20T13:34:11Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Introduction to dynamical mean-field theory of generic random neural networks [2.0711789781518752]
  It is not easy for beginners to access the essence of this tool and the underlying physics. We give a pedagogical introduction of this method in a particular example of generic random neural networks. The numerical implementation of solving the integro-differential mean-field equations is also detailed.
  arXiv  Detail & Related papers  (2023-05-15T09:01:40Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• Training Integrable Parameterizations of Deep Neural Networks in the Infinite-Width Limit [0.0]
  Large-width dynamics has emerged as a fruitful viewpoint and led to practical insights on real-world deep networks. For two-layer neural networks, it has been understood that the nature of the trained model radically changes depending on the scale of the initial random weights. We propose various methods to avoid this trivial behavior and analyze in detail the resulting dynamics.
  arXiv  Detail & Related papers  (2021-10-29T07:53:35Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation [76.57464214864756]
  We study function approximation with two-layer neural networks. Our results significantly improve upon what can be attained with linear (or eluder dimension) methods.
  arXiv  Detail & Related papers  (2021-07-14T03:03:56Z)
• Shallow Representation is Deep: Learning Uncertainty-aware and Worst-case Random Feature Dynamics [1.1470070927586016]
  This paper views uncertain system models as unknown or uncertain smooth functions in universal kernel Hilbert spaces. By directly approximating the one-step dynamics function using random features with uncertain parameters, we then view the whole dynamical system as a multi-layer neural network.
  arXiv  Detail & Related papers  (2021-06-24T14:48:12Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• Deep Randomized Neural Networks [12.333836441649343]
  Randomized Neural Networks explore the behavior of neural systems where the majority of connections are fixed. This chapter surveys all the major aspects regarding the design and analysis of Randomized Neural Networks.
  arXiv  Detail & Related papers  (2020-02-27T17:57:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.