Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks

• URL: http://arxiv.org/abs/2505.09816v1
• Date: Wed, 14 May 2025 21:35:55 GMT
• Title: Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks
• Authors: Yi Xie, Stefan Mihalas, Łukasz Kuśmierz,
• Abstract summary: We study the activity of recurrent neural networks (RNNs) with random weights drawn from biologically plausible L'evy alpha-stable distributions.<n>We theoretically predict the gain at which the system transitions from quiescent to chaotic dynamics, and validate it through simulations.<n>Our results reveal a biologically aligned tradeoff between the robustness of dynamics near the edge of chaos and the richness of high-dimensional neural activity.
• Score: 4.098619171200725
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Growing evidence suggests that synaptic weights in the brain follow heavy-tailed distributions, yet most theoretical analyses of recurrent neural networks (RNNs) assume Gaussian connectivity. We systematically study the activity of RNNs with random weights drawn from biologically plausible L\'evy alpha-stable distributions. While mean-field theory for the infinite system predicts that the quiescent state is always unstable -- implying ubiquitous chaos -- our finite-size analysis reveals a sharp transition between quiescent and chaotic dynamics. We theoretically predict the gain at which the system transitions from quiescent to chaotic dynamics, and validate it through simulations. Compared to Gaussian networks, heavy-tailed RNNs exhibit a broader parameter regime near the edge of chaos, namely a slow transition to chaos. However, this robustness comes with a tradeoff: heavier tails reduce the Lyapunov dimension of the attractor, indicating lower effective dimensionality. Our results reveal a biologically aligned tradeoff between the robustness of dynamics near the edge of chaos and the richness of high-dimensional neural activity. By analytically characterizing the transition point in finite-size networks -- where mean-field theory breaks down -- we provide a tractable framework for understanding dynamics in realistically sized, heavy-tailed neural circuits.

Related papers

• Langevin Flows for Modeling Neural Latent Dynamics [81.81271685018284]
  We introduce LangevinFlow, a sequential Variational Auto-Encoder where the time evolution of latent variables is governed by the underdamped Langevin equation.<n>Our approach incorporates physical priors -- such as inertia, damping, a learned potential function, and forces -- to represent both autonomous and non-autonomous processes in neural systems.<n>Our method outperforms state-of-the-art baselines on synthetic neural populations generated by a Lorenz attractor.
  arXiv  Detail & Related papers  (2025-07-15T17:57:48Z)
• Sparse chaos in cortical circuits [3.4137115855910767]
  We show that basic features of nerve impulse generation profoundly affect collective chaos in neuronal circuits.<n>We find a drastic reduction in the number of unstable manifold, Kolmogorov-Sinai entropy, and attractor dimension.<n>In cortical circuits, biophysical properties appear tuned to this regime of sparse chaos.
  arXiv  Detail & Related papers  (2024-12-30T18:55:35Z)
• Exploiting Chaotic Dynamics as Deep Neural Networks [1.9282110216621833]
  We show that the essence of chaos can be found in various state-of-the-art deep neural networks. Our framework presents superior results in terms of accuracy, convergence speed, and efficiency. This study offers a new path for the integration of chaos, which has long been overlooked in information processing.
  arXiv  Detail & Related papers  (2024-05-29T22:03:23Z)
• Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
  We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence. We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers. This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
  arXiv  Detail & Related papers  (2023-07-20T13:34:11Z)
• Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks [0.8932296777085644]
  Conventional artificial deep neural networks operating near the phase boundary of the signal propagation dynamics, also known as the edge of chaos, exhibit universal scaling laws of absorbing phase transitions.<n>We exploit the fully deterministic nature of the propagation dynamics to elucidate an analogy between a signal collapse in the neural networks and an absorbing state.
  arXiv  Detail & Related papers  (2023-07-05T13:39:02Z)
• Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network [0.0]
  Despite being observed in the brain and in artificial networks, the mechanisms of drift and its implications are not fully understood. Motivated by recent experimental findings of stimulus-dependent drift in the piriform cortex, we use theory and simulations to study this phenomenon in a two-layer linear feedforward network.
  arXiv  Detail & Related papers  (2023-02-06T04:56:05Z)
• Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks [59.142826407441106]
  We study the generalization behavior of shallow neural networks (SNNs) by leveraging the concept of algorithmic stability. We consider gradient descent (GD) and gradient descent (SGD) to train SNNs, for both of which we develop consistent excess bounds.
  arXiv  Detail & Related papers  (2022-09-19T18:48:00Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• Extended critical regimes of deep neural networks [0.0]
  We show that heavy-tailed weights enable the emergence of an extended critical regime without fine-tuning parameters. In this extended critical regime, DNNs exhibit rich and complex propagation dynamics across layers. We provide a theoretical guide for the design of efficient neural architectures.
  arXiv  Detail & Related papers  (2022-03-24T10:15:50Z)
• Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics [50.83356836818667]
  We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
  arXiv  Detail & Related papers  (2020-07-11T18:19:50Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.