How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

• URL: http://arxiv.org/abs/2605.05113v1
• Date: Wed, 06 May 2026 16:44:44 GMT
• Title: How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences
• Abstract summary: We study signal propagation in linear recurrent models at finite width.<n>We find that finite-width effects accumulate more rapidly with depth in recurrent models than in feedforward ones.
• Score: 2.335152769484957
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study signal propagation in linear recurrent models at finite width. While existing signal propagation theory relies predominantly on the infinite-width limit, it remains unclear for how long that approximation remains accurate when recurrent depth $t$ grows jointly with width $n$. This question is especially relevant for modern recurrent sequence models, whose natural operating regime involves long input sequences, i.e., large $t$. We derive exact finite-width formulas for the hidden state signal energies in linear recurrences under complex Gaussian initialization. Using these formulas, we identify the joint depth-width scaling regimes that govern signal propagation: (i) a subcritical regime $t=o(\sqrt n)$, in which the infinite-width approximation remains valid; (ii) a critical regime $t\sim c\sqrt n$, in which non-negligible deviations from infinite-width predictions appear and a nontrivial joint scaling limit emerges; and (iii) a supercritical regime $t\gg \sqrt n$, in which finite-width effects dominate. Thus, our results pinpoint the precise recurrent depth scale at which infinite-width theory breaks down in long-range linear recurrences. In turn, this shows when standard initialization schemes, such as Glorot, become unstable. More broadly, our results demonstrate that finite-width effects accumulate more rapidly with depth in recurrent models than in feedforward ones, leading to qualitatively different signal propagation behavior.

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer [46.50233782149623]
  We study the evolution of hidden-weight spectra in wide neural networks trained by (stochastic) gradient descent.<n>We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for spiked ensembles.
  arXiv  Detail & Related papers  (2026-05-08T15:28:01Z)
• Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networks [21.223229143122897]
  We develop the first emphfully initialization-dependent bounds for shallow neural networks with general Lipschitz activation functions.<n>Our bounds depend on the path-norm of the distance from initialization, which are derived by introducing a new peeling technique.<n>We also develop a lower bound tight up to a constant factor.
  arXiv  Detail & Related papers  (2026-04-01T05:42:40Z)
• Parallel Complex Diffusion for Scalable Time Series Generation [50.01609741902786]
  PaCoDi is a spectral-native architecture that decouples generative modeling in the frequency domain.<n>We show that PaCoDi outperforms existing baselines in both generation quality and inference speed.
  arXiv  Detail & Related papers  (2026-02-10T14:31:53Z)
• Revisiting Glorot Initialization for Long-Range Linear Recurrences [35.51491162256673]
  Glorot is designed to ensure stable signal propagation.<n>Small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode.<n>We propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one.
  arXiv  Detail & Related papers  (2025-05-26T11:04:59Z)
• Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability [69.01076284478151]
  In machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime.
  arXiv  Detail & Related papers  (2023-05-19T16:24:47Z)
• Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
  Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We show that linear networks make provably optimal predictions at infinite depth. We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
  arXiv  Detail & Related papers  (2022-12-29T20:57:46Z)
• High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Models [3.712310463095685]
  We provide a "path-wise" characterization of the overlap between the output of the Langevin algorithm and the planted signal. As a second contribution, we derive an explicit formula for the limiting overlap in terms of the signal-to-noise ratio and the injected noise in the diffusion.
  arXiv  Detail & Related papers  (2022-04-09T14:03:21Z)
• The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization [18.613475245655806]
  We study ReLU ResNets in the infinite-depth-and-width limit, where both depth and width tend to infinity as their ratio, $d/n$, remains constant. Using Monte Carlo simulations, we demonstrate that even basic properties of standard ResNet architectures are poorly captured by the Gaussian limit.
  arXiv  Detail & Related papers  (2021-06-07T23:47:37Z)
• Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems [24.229336600210015]
  Recently, depth separation results for ReLU networks were obtained via a new connection with dynamical systems. We improve the existing width lower bounds along several aspects. A byproduct of our results is that there exists a universal constant characterizing the depth-width trade-offs.
  arXiv  Detail & Related papers  (2020-03-02T11:36:26Z)
• On Random Kernels of Residual Architectures [93.94469470368988]
  We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets. Our findings show that in ResNets, convergence to the NTK may occur when depth and width simultaneously tend to infinity. In DenseNets, however, convergence of the NTK to its limit as the width tends to infinity is guaranteed.
  arXiv  Detail & Related papers  (2020-01-28T16:47:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.