Generalization at the Edge of Stability

• URL: http://arxiv.org/abs/2604.19740v1
• Date: Tue, 21 Apr 2026 17:59:02 GMT
• Title: Generalization at the Edge of Stability
• Abstract summary: Training modern neural networks often relies on large learning rates, operating at the edge of stability.<n> Empirically, this regime often yields improved generalization, yet the underlying mechanism remains poorly understood.
• Score: 19.235085742973418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training modern neural networks often relies on large learning rates, operating at the edge of stability, where the optimization dynamics exhibit oscillatory and chaotic behavior. Empirically, this regime often yields improved generalization performance, yet the underlying mechanism remains poorly understood. In this work, we represent stochastic optimizers as random dynamical systems, which often converge to a fractal attractor set (rather than a point) with a smaller intrinsic dimension. Building on this connection and inspired by Lyapunov dimension theory, we introduce a novel notion of dimension, coined the `sharpness dimension', and prove a generalization bound based on this dimension. Our results show that generalization in the chaotic regime depends on the complete Hessian spectrum and the structure of its partial determinants, highlighting a complexity that cannot be captured by the trace or spectral norm considered in prior work. Experiments across various MLPs and transformers validate our theory while also providing new insights into the recently observed phenomenon of grokking.

Related papers

• Fourier fractal dimension to predict the generalization of deep neural networks [0.0]
  We propose a novel generalization measure based on the Fourier fractal dimension of the network's weight variations.<n>By analyzing the characteristic function of the Lévy-driven differential equations in the frequency domain, we extract a metric that robustly captures the geometric complexity of the learning process.<n>Our method achieves state-of-the-art Kendall rank correlation, outperforming a wide array of existing norm-based, margin-based, and PAC-Bayesian measures.
  arXiv  Detail & Related papers  (2026-06-06T19:40:54Z)
• The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization [57.37943479039033]
  We study how architectural inductive bias reshapes the implicit regularization induced by the edge-of-stability phenomenon in gradient descent.<n>We show that locality and weight sharing fundamentally change this picture.
  arXiv  Detail & Related papers  (2026-03-05T04:50:51Z)
• KoopGen: Koopman Generator Networks for Representing and Predicting Dynamical Systems with Continuous Spectra [65.11254608352982]
  We introduce a generator-based neural Koopman framework that models dynamics through a structured, state-dependent representation of Koopman generators.<n>By exploiting the intrinsic Cartesian decomposition into skew-adjoint and self-adjoint components, KoopGen separates conservative transport from irreversible dissipation.
  arXiv  Detail & Related papers  (2026-02-15T06:32:23Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• Diagnosing chaos with projected ensembles of process tensors [0.22499166814992436]
  We introduce the projected process ensemble an ensemble of pure states of a process tensor in a given basis of local interventions, and use to define increasingly more fine-grained probes of quantum chaos.<n>We discover characteristic entanglement structures within the ensemble that can distinguish sharply chaotic from integrable dynamics, overcoming deficiencies of the quantum dynamical and entemporaltropies.<n>Our work elucidates the fingerprints of chaos in interacting quantum processes, and provides a unified framework for analyzing unitary and monitored many-body dynamics.
  arXiv  Detail & Related papers  (2025-02-19T18:06:07Z)
• Approaching Deep Learning through the Spectral Dynamics of Weights [41.948042468042374]
  spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to clarify and unify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale grokking'' to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers.
  arXiv  Detail & Related papers  (2024-08-21T17:48:01Z)
• Generalization of Scaled Deep ResNets in the Mean-Field Regime [55.77054255101667]
  We investigate emphscaled ResNet in the limit of infinitely deep and wide neural networks. Our results offer new insights into the generalization ability of deep ResNet beyond the lazy training regime.
  arXiv  Detail & Related papers  (2024-03-14T21:48:00Z)
• PAC-Chernoff Bounds: Understanding Generalization in the Interpolation Regime [6.645111950779666]
  This paper introduces a distribution-dependent PAC-Chernoff bound that exhibits perfect tightness for interpolators.<n>We present a unified theoretical framework revealing why certain interpolators show an exceptional generalization, while others falter.
  arXiv  Detail & Related papers  (2023-06-19T14:07:10Z)
• On the generalization of learning algorithms that do not converge [54.122745736433856]
  Generalization analyses of deep learning typically assume that the training converges to a fixed point. Recent results indicate that in practice, the weights of deep neural networks optimized with gradient descent often oscillate indefinitely.
  arXiv  Detail & Related papers  (2022-08-16T21:22:34Z)
• On dissipative symplectic integration with applications to gradient-based optimization [77.34726150561087]
  We propose a geometric framework in which discretizations can be realized systematically. We show that a generalization of symplectic to nonconservative and in particular dissipative Hamiltonian systems is able to preserve rates of convergence up to a controlled error.
  arXiv  Detail & Related papers  (2020-04-15T00:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.