Universality and approximation bounds for echo state networks with random weights

• URL: http://arxiv.org/abs/2206.05669v3
• Date: Mon, 4 Dec 2023 02:05:25 GMT
• Title: Universality and approximation bounds for echo state networks with random weights
• Authors: Zhen Li, Yunfei Yang
• Abstract summary: We study the uniform approximation of echo state networks with randomly generated internal weights. Recent results showed that echo state networks with ReLU activation are universal.
• Score: 5.310194156119441
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the uniform approximation of echo state networks with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that echo state networks with ReLU activation are universal. In this paper, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the echo state network can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU echo state networks for sufficiently regular operators.

Related papers

• Contraction, Criticality, and Capacity: A Dynamical-Systems Perspective on Echo-State Networks [13.857230672081489]
  We present a unified, dynamical-systems treatment that weaves together functional analysis, random attractor theory and recent neuroscientific findings.<n>First, we prove that the Echo-State Property (wash-out of initial conditions) together with global Lipschitz dynamics necessarily yields the Fading-Memory Property.<n>Second, employing a Stone-Weierstrass strategy we give a streamlined proof that ESNs with nonlinear reservoirs and linear read-outs are dense in the Banach space of causal, time-in fading-memory filters.<n>Third, we quantify computational resources via memory-capacity spectrum, show how
  arXiv  Detail & Related papers  (2025-07-24T14:41:18Z)
• Universal Approximation Theorem for Deep Q-Learning via FBSDE System [2.1756081703276]
  This paper establishes a Universal Approximation Theorem for a class of Deep Q-Networks (DQNs)<n>We show that layers of a deep residual network, conceived as neural operators acting on function spaces, can approximate the action of the Bellman operator.
  arXiv  Detail & Related papers  (2025-05-09T13:11:55Z)
• The Spectral Bias of Shallow Neural Network Learning is Shaped by the Choice of Non-linearity [0.7499722271664144]
  We study how non-linear activation functions contribute to shaping neural networks' implicit bias. We show that local dynamical attractors facilitate the formation of clusters of hyperplanes where the input to a neuron's activation function is zero.
  arXiv  Detail & Related papers  (2025-03-13T17:36:46Z)
• Efficient Training of Deep Neural Operator Networks via Randomized Sampling [0.0]
  Deep operator network (DeepNet) has demonstrated success in the real-time prediction of complex dynamics across various scientific and engineering applications. We introduce a random sampling technique to be adopted the training of DeepONet, aimed at improving generalization ability of the model, while significantly reducing computational time. Our results indicate that incorporating randomization in the trunk network inputs during training enhances the efficiency and robustness of DeepONet, offering a promising avenue for improving the framework's performance in modeling complex physical systems.
  arXiv  Detail & Related papers  (2024-09-20T07:18:31Z)
• Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control [63.310780486820796]
  We show how a parameterization of recurrent connectivity influences robustness in closed-loop settings. We find that closed-form continuous-time neural networks (CfCs) with fewer parameters can outperform their full-rank, fully-connected counterparts.
  arXiv  Detail & Related papers  (2023-10-05T21:44:18Z)
• Quantum correlation functions through tensor network path integral [0.0]
  tensor networks are utilized for calculating equilibrium correlation function for open quantum systems. The influence of the solvent on the quantum system is incorporated through an influence functional. The design and implementation of this method is discussed along with illustrations from rate theory, symmetrized spin correlation functions, dynamical susceptibility calculations and quantum thermodynamics.
  arXiv  Detail & Related papers  (2023-08-21T07:46:51Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control [0.0]
  Recent results have demonstrated that for specialized activation functions, such as ReLUs, high accuracy can be achieved via linear combinations of randomly activations. This paper defines mollified integral representations, which provide a means to form integral representations of target functions using activations for which no direct integral representation is currently known.
  arXiv  Detail & Related papers  (2023-03-28T18:55:48Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• The efficacy and generalizability of conditional GANs for posterior inference in physics-based inverse problems [0.4588028371034407]
  We train conditional Wasserstein generative adversarial networks to effectively sample from the posterior of physics-based Bayesian inference problems. We show the generator can learn inverse maps which are local in nature, which in turn promotes generalizability when testing with out-of-distribution samples.
  arXiv  Detail & Related papers  (2022-02-15T22:57:05Z)
• UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
  We provide a proof that a class of learnable functions can universally approximate any valid intensity function. We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
  arXiv  Detail & Related papers  (2020-07-28T09:31:56Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• Approximation Bounds for Random Neural Networks and Reservoir Systems [8.143750358586072]
  This work studies approximation based on single-hidden-layer feedforward and recurrent neural networks with randomly generated internal weights. In particular, this proves that echo state networks with randomly generated weights are capable of approximating a wide class of dynamical systems arbitrarily well.
  arXiv  Detail & Related papers  (2020-02-14T09:43:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.