Analyzing Echo-state Networks Using Fractal Dimension

• URL: http://arxiv.org/abs/2205.09348v2
• Date: Thu, 26 May 2022 17:00:06 GMT
• Title: Analyzing Echo-state Networks Using Fractal Dimension
• Authors: Norbert Michael Mayer, Oliver Obst
• Abstract summary: We build on the observation that input sequences appear as fractal patterns in their hidden state representation. These patterns have a fractal dimension that is lower than the number of units in the reservoir.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work joins aspects of reservoir optimization, information-theoretic optimal encoding, and at its center fractal analysis. We build on the observation that, due to the recursive nature of recurrent neural networks, input sequences appear as fractal patterns in their hidden state representation. These patterns have a fractal dimension that is lower than the number of units in the reservoir. We show potential usage of this fractal dimension with regard to optimization of recurrent neural network initialization. We connect the idea of `ideal' reservoirs to lossless optimal encoding using arithmetic encoders. Our investigation suggests that the fractal dimension of the mapping from input to hidden state shall be close to the number of units in the network. This connection between fractal dimension and network connectivity is an interesting new direction for recurrent neural network initialization and reservoir computing.

Related papers

• On the Dimension of Pullback Attractors in Recurrent Neural Networks [0.0]
  Recently, it has been conjectured that reservoir computers, a particular class of RNNs, trained on observations of a dynamical systems can be interpreted as embeddings. In this work, we use a nonautonomous dynamical systems approach to establish an upper bound for the fractal dimension of the subset of reservoir state space approximated during training and prediction phase.
  arXiv  Detail & Related papers  (2025-01-20T09:38:30Z)
• On The Potential of The Fractal Geometry and The CNNs Ability to Encode it [1.7311053765541484]
  The fractal dimension provides a statistical index of object complexity. Although useful in several classification tasks, the fractal dimension is under-explored in deep learning applications. We show that training a shallow network on fractal features achieves performance comparable to that of deep networks trained on raw data.
  arXiv  Detail & Related papers  (2024-01-07T15:22:56Z)
• Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
  Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We show that linear networks make provably optimal predictions at infinite depth. We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
  arXiv  Detail & Related papers  (2022-12-29T20:57:46Z)
• Learning Neural Volumetric Field for Point Cloud Geometry Compression [13.691147541041804]
  We propose to code the geometry of a given point cloud by learning a neural field. We divide the entire space into small cubes and represent each non-empty cube by a neural network and an input latent code. The network is shared among all the cubes in a single frame or multiple frames, to exploit the spatial and temporal redundancy.
  arXiv  Detail & Related papers  (2022-12-11T19:55:24Z)
• Dense Hebbian neural networks: a replica symmetric picture of supervised learning [4.133728123207142]
  We consider dense, associative neural-networks trained by a teacher with supervision. We investigate their computational capabilities analytically, via statistical-mechanics of spin glasses, and numerically, via Monte Carlo simulations.
  arXiv  Detail & Related papers  (2022-11-25T13:37:47Z)
• NAF: Neural Attenuation Fields for Sparse-View CBCT Reconstruction [79.13750275141139]
  This paper proposes a novel and fast self-supervised solution for sparse-view CBCT reconstruction. The desired attenuation coefficients are represented as a continuous function of 3D spatial coordinates, parameterized by a fully-connected deep neural network. A learning-based encoder entailing hash coding is adopted to help the network capture high-frequency details.
  arXiv  Detail & Related papers  (2022-09-29T04:06:00Z)
• A singular Riemannian geometry approach to Deep Neural Networks II. Reconstruction of 1-D equivalence classes [78.120734120667]
  We build the preimage of a point in the output manifold in the input space. We focus for simplicity on the case of neural networks maps from n-dimensional real spaces to (n - 1)-dimensional real spaces.
  arXiv  Detail & Related papers  (2021-12-17T11:47:45Z)
• A Sparse Coding Interpretation of Neural Networks and Theoretical Implications [0.0]
  Deep convolutional neural networks have achieved unprecedented performance in various computer vision tasks. We propose a sparse coding interpretation of neural networks that have ReLU activation. We derive a complete convolutional neural network without normalization and pooling.
  arXiv  Detail & Related papers  (2021-08-14T21:54:47Z)
• Towards Efficient Graph Convolutional Networks for Point Cloud Handling [181.59146413326056]
  We aim at improving the computational efficiency of graph convolutional networks (GCNs) for learning on point clouds. A series of experiments show that optimized networks have reduced computational complexity, decreased memory consumption, and accelerated inference speed.
  arXiv  Detail & Related papers  (2021-04-12T17:59:16Z)
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks [56.084798078072396]
  We take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time. We show that convergence to a global minimum is guaranteed for networks with quadratic widths in the sample size and linear in their depth at a time logarithmic in both. Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size.
  arXiv  Detail & Related papers  (2021-01-12T00:40:45Z)
• MSE-Optimal Neural Network Initialization via Layer Fusion [68.72356718879428]
  Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. The use of gradient combined nonvolutionity renders learning susceptible to novel problems. We propose fusing neighboring layers of deeper networks that are trained with random variables.
  arXiv  Detail & Related papers  (2020-01-28T18:25:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.