Refined Kolmogorov Complexity of Analog, Evolving and Stochastic Recurrent Neural Networks

• URL: http://arxiv.org/abs/2309.17032v1
• Date: Fri, 29 Sep 2023 07:38:50 GMT
• Title: Refined Kolmogorov Complexity of Analog, Evolving and Stochastic Recurrent Neural Networks
• Authors: J\'er\'emie Cabessa, Yann Strozecki
• Abstract summary: We provide a refined characterization of the super-Turing computational power of analog, evolving, and neural networks based on the Kolmogorov complexity of their real weights, evolving weights, and real probabilities, respectively.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We provide a refined characterization of the super-Turing computational power of analog, evolving, and stochastic neural networks based on the Kolmogorov complexity of their real weights, evolving weights, and real probabilities, respectively. First, we retrieve an infinite hierarchy of classes of analog networks defined in terms of the Kolmogorov complexity of their underlying real weights. This hierarchy is located between the complexity classes $\mathbf{P}$ and $\mathbf{P/poly}$. Then, we generalize this result to the case of evolving networks. A similar hierarchy of Kolomogorov-based complexity classes of evolving networks is obtained. This hierarchy also lies between $\mathbf{P}$ and $\mathbf{P/poly}$. Finally, we extend these results to the case of stochastic networks employing real probabilities as source of randomness. An infinite hierarchy of stochastic networks based on the Kolmogorov complexity of their probabilities is therefore achieved. In this case, the hierarchy bridges the gap between $\mathbf{BPP}$ and $\mathbf{BPP/log^*}$. Beyond proving the existence and providing examples of such hierarchies, we describe a generic way of constructing them based on classes of functions of increasing complexity. For the sake of clarity, this study is formulated within the framework of echo state networks. Overall, this paper intends to fill the missing results and provide a unified view about the refined capabilities of analog, evolving and stochastic neural networks.

Related papers

• When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective [22.831594980764216]
  We prove that feedforward and recurrent neural networks may suffer from larger sample complexity compared to Transformers. Our proposed sparse retrieval model illustrates a natural hierarchy in sample complexity across these architectures.
  arXiv  Detail & Related papers  (2025-03-14T10:30:42Z)
• Pruning is Optimal for Learning Sparse Features in High-Dimensions [15.967123173054535]
  We show that a class of statistical models can be optimally learned using pruned neural networks trained with gradient descent. We show that pruning neural networks proportional to the sparsity level of $boldsymbolV$ improves their sample complexity compared to unpruned networks.
  arXiv  Detail & Related papers  (2024-06-12T21:43:12Z)
• Defining Neural Network Architecture through Polytope Structures of Dataset [53.512432492636236]
  This paper defines upper and lower bounds for neural network widths, which are informed by the polytope structure of the dataset in question. We develop an algorithm to investigate a converse situation where the polytope structure of a dataset can be inferred from its corresponding trained neural networks. It is established that popular datasets such as MNIST, Fashion-MNIST, and CIFAR10 can be efficiently encapsulated using no more than two polytopes with a small number of faces.
  arXiv  Detail & Related papers  (2024-02-04T08:57:42Z)
• Structural Balance and Random Walks on Complex Networks with Complex Weights [13.654842079699458]
  Recent years have seen an increasing interest to extend the tools of network science when the weight of edges are complex numbers. Here, we focus on the case when the weight matrix is Hermitian, a reasonable assumption in many applications. We introduce a classification of complex-weighted networks based on the notion of structural balance, and illustrate the shared spectral properties within each type.
  arXiv  Detail & Related papers  (2023-07-04T16:39:52Z)
• Polyhedral Complex Extraction from ReLU Networks using Edge Subdivision [0.0]
  A neural network consists of piecewise affine building blocks, such as fully-connected layers and ReLU activations. This complex has been previously studied to characterize theoretical properties of neural networks. We propose to subdivide the regions via intersections with hyperplanes induced by each neuron.
  arXiv  Detail & Related papers  (2023-06-12T16:17:04Z)
• Computational Complexity of Learning Neural Networks: Smoothness and Degeneracy [52.40331776572531]
  We show that learning depth-$3$ ReLU networks under the Gaussian input distribution is hard even in the smoothed-analysis framework. Our results are under a well-studied assumption on the existence of local pseudorandom generators.
  arXiv  Detail & Related papers  (2023-02-15T02:00:26Z)
• Dist2Cycle: A Simplicial Neural Network for Homology Localization [66.15805004725809]
  Simplicial complexes can be viewed as high dimensional generalizations of graphs that explicitly encode multi-way ordered relations. We propose a graph convolutional model for learning functions parametrized by the $k$-homological features of simplicial complexes.
  arXiv  Detail & Related papers  (2021-10-28T14:59:41Z)
• Dive into Layers: Neural Network Capacity Bounding using Algebraic Geometry [55.57953219617467]
  We show that the learnability of a neural network is directly related to its size. We use Betti numbers to measure the topological geometric complexity of input data and the neural network. We perform the experiments on a real-world dataset MNIST and the results verify our analysis and conclusion.
  arXiv  Detail & Related papers  (2021-09-03T11:45:51Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces [208.67848059021915]
  We study the exploration-exploitation tradeoff at the core of reinforcement learning. In particular, we prove that the complexity of the function class $mathcalF$ characterizes the complexity of the function. Our regret bounds are independent of the number of episodes.
  arXiv  Detail & Related papers  (2020-11-09T18:32:22Z)
• Consistency of Spectral Clustering on Hierarchical Stochastic Block Models [5.983753938303726]
  We study the hierarchy of communities in real-world networks under a generic block model. We prove the strong consistency of this method under a wide range of model parameters. Unlike most of existing work, our theory covers multiscale networks where the connection probabilities may differ by orders of magnitude.
  arXiv  Detail & Related papers  (2020-04-30T01:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.