Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks

• URL: http://arxiv.org/abs/2606.29519v1
• Date: Sun, 28 Jun 2026 17:30:36 GMT
• Title: Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks
• Abstract summary: We show that long decay class $f(ell)$ is not fixed by the architecture.<n>It emerges from the coupling between the state dynamics and parameter dynamics.<n>A single exponent governs both the spread of time scales and how slowly the network forgets.
• Score: 3.924071936547547
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-range learning is hard for recurrent networks trained with stochastic gradient descent, because the influence of a past input fades with the lag $\ell$, and if it fades too fast the dependence cannot be learned from finite data. This fade is captured by an envelope $f(\ell)$. An exponential fade makes the data needed to learn a lag-$\ell$ dependence grow exponentially, putting long horizons out of reach; a power-law fade keeps the cost polynomial. We show that the asymptotic decay class of $f(\ell)$ is not fixed by the architecture. Instead, it emerges from the coupling between the state dynamics and parameter dynamics, settling into either a collapsed regime (fast, exponential forgetting) or an extended, anti-collapsed regime (slow, power-law forgetting). The intuition is a competition within these coupled dynamics. Training drives the network's effective time scales toward short ones, while rare, heavy-tailed fluctuations of the learning dynamics push a few of them to very long values. The extended regime survives only when these heavy-tailed pushes are strong enough to balance the pull. We make this mathematically precise with a coarse-grained stochastic process and prove exactly when the extended regime exists. A single exponent, the spectral exponent~$β$, then governs both the spread of time scales and how slowly the network forgets. Realizing the regime in practice needs one more ingredient: the joint action of the architecture and the optimizer must be able to hold such a broad spread. A network whose capacity to generate broad time-scale spectra is severely constrained still collapses, even when supplied with strong heavy-tailed forcing. Heavy-tailed fluctuations thus act not as noise to be suppressed, but as the mechanism that sustains long-range learning.

Related papers

• Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions [5.674162165646654]
  Existing theory of momentum assumes that gradients arrive at every parameter at a roughly constant rate.<n>We theoretically analyze the dynamics of two tractable models of momentum under sparse updates.
  arXiv  Detail & Related papers  (2026-05-27T18:06:41Z)
• Renormalizable Spectral-Shell Dynamics as the Origin of Neural Scaling Laws [2.779943773196378]
  We show that deep-network training obeys a simple macroscopic structure despite highly nonlinear optimization dynamics.<n>For mean-squared error loss, the training error evolves as $dot e_t=-M(t)e_t$ with $M(t)=J_(t)J_(t)!*$, a time-dependent self-adjoint operator induced by the network Jacobian.<n>This framework explains neural scaling laws and double descent, and unifies lazy (NTK-like) training and feature learning as two limits of the same spectral-shell
  arXiv  Detail & Related papers  (2025-12-11T08:38:46Z)
• Learnability Window in Gated Recurrent Neural Networks [3.924071936547547]
  Gate mechanisms determine the learnability window $mathcalH_N$ of recurrent neural networks.<n>We show that learnability is governed by the empheffective learning rates $_t,ell$, per-lag and per-neuron quantities.
  arXiv  Detail & Related papers  (2025-12-05T15:16:59Z)
• Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks [59.552873049024775]
  We show that compute-optimally trained models exhibit a remarkably precise universality.<n>With learning rate decay, the collapse becomes so tight that differences in the normalized curves across models fall below the noise floor.<n>We explain these phenomena by connecting collapse to the power-law structure in typical neural scaling laws.
  arXiv  Detail & Related papers  (2025-07-02T20:03:34Z)
• Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets [58.460298576330835]
  We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth'<n>We leverage this intuition to explain the emergence of a bottleneck structure, as observed in previous work.
  arXiv  Detail & Related papers  (2024-05-27T18:15:05Z)
• Super Consistency of Neural Network Landscapes and Learning Rate Transfer [72.54450821671624]
  We study the landscape through the lens of the loss Hessian. We find that certain spectral properties under $mu$P are largely independent of the size of the network. We show that in the Neural Tangent Kernel (NTK) and other scaling regimes, the sharpness exhibits very different dynamics at different scales.
  arXiv  Detail & Related papers  (2024-02-27T12:28:01Z)
• Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks [47.73646927060476]
  We analyze the dynamics of finite width effects in wide but finite feature learning neural networks. Our results are non-perturbative in the strength of feature learning.
  arXiv  Detail & Related papers  (2023-04-06T23:11:49Z)
• Scaling Laws Beyond Backpropagation [64.0476282000118]
  We study the ability of Direct Feedback Alignment to train causal decoder-only Transformers efficiently. We find that DFA fails to offer more efficient scaling than backpropagation.
  arXiv  Detail & Related papers  (2022-10-26T10:09:14Z)
• Deep Explicit Duration Switching Models for Time Series [84.33678003781908]
  We propose a flexible model that is capable of identifying both state- and time-dependent switching dynamics. State-dependent switching is enabled by a recurrent state-to-switch connection. An explicit duration count variable is used to improve the time-dependent switching behavior.
  arXiv  Detail & Related papers  (2021-10-26T17:35:21Z)
• Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem [8.492084752803528]
  Given a target function $f$, how large must a neural network be in order to approximate $f$? Recent works examine this basic question on neural network textitexpressivity'' from the lens of dynamical systems.
  arXiv  Detail & Related papers  (2021-10-19T22:28:27Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)
• Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems [24.229336600210015]
  Recently, depth separation results for ReLU networks were obtained via a new connection with dynamical systems. We improve the existing width lower bounds along several aspects. A byproduct of our results is that there exists a universal constant characterizing the depth-width trade-offs.
  arXiv  Detail & Related papers  (2020-03-02T11:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.