The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow

• URL: http://arxiv.org/abs/2609.01034v1
• Date: Tue, 01 Sep 2026 10:32:48 GMT
• Title: The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow
• Abstract summary: We derive a perturbative regime in which the central flow is the limit of gradient descent.<n>Three timescales emerge: a fast timescale of oscillations along the sharpest direction, an intermediate timescale of the self-stabilization mechanism, and a slow timescale of the dynamics along the minimizers of $g$-the central flow.
• Score: 8.426855646402238
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The central flow of Cohen et al. (2025) is an empirically accurate continuous-time model of gradient descent at the edge of stability in deep learning, However, its derivation is heuristic. We propose a perturbative regime in which the central flow is the limit of gradient descent: we assume that the loss decomposes as $f = g + \varepsilon h$; in the limit $\varepsilon \to 0$, the dynamics of gradient descent with learning rate $η$ converge to the gradient flow of $h$ constrained to the minimizers of $g$ of sharpness at most $2/η$. Our approach is formal rather than rigorous; it treats gradient descent as a singularly perturbed dynamical system in $\varepsilon$. Three timescales emerge: a fast timescale of oscillations along the sharpest direction, an intermediate timescale of the self-stabilization mechanism, and a slow timescale of the dynamics along the minimizers of $g$-the central flow. Using the method of multiple scales, a classical formal method from singular perturbation theory, we derive the expansion of the dynamics in $\varepsilon$: the central flow emerges as the leading-order term in the expansion, while the self-stabilization mechanism appears in the next-order term. We study this mechanism beyond previous analyses: with a single eigenvalue at the edge of stability, we compute the slow drift of the energy of the fluctuations; with several eigenvalues at the edge of stability, we derive the self-stabilization system and explain why fluctuations persist.

Related papers

• Strong Averaging Principle and Long-Time Dynamics for Fast-Slow SDEs with Increasing Time-Scale Separation and Degenerate Noise [1.0026496861838448]
  We establish a strong averaging principle for fast-slow differential equations with a time-dependent scale-separation parameter.<n>In contrast to approaches based on noise-induced smoothing or regularity, our approach relies on dissipativity of the frozen fast dynamics.
  arXiv  Detail & Related papers  (2026-08-24T16:30:59Z)
• The Road Taken: The Role of Optimizers at the Edge of Stability [53.81268610971847]
  edge of stability refers to a phenomenon in deep learning where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent predicts to be unstable.<n>Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate.<n>We observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as $times 21.1$.
  arXiv  Detail & Related papers  (2026-08-19T01:03:34Z)
• Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability [11.598128347617957]
  We propose a tractable, faithful, and predictive model of gradient descent dynamics at the edge of stability (EoS)<n>We show that Edge Flow tracks the dynamics of gradient descent at least as faithfully as previously proposed continuous-time EoS models.
  arXiv  Detail & Related papers  (2026-06-16T15:45:47Z)
• SGD at the Edge of Stability: The Stochastic Sharpness Gap [10.176501817419371]
  When training networks with full-batch gradient descent (GD) and step size $$, the largest eigenvalue of the Hessian rises to $2/$ and hovers there, a phenomenon termed the Edge of Stability (EoS)<n>citetdamian2023selfstab showed that this behavior is explained by a self-stabilization mechanism driven by third-order structure of the loss, and that GD implicitly follows projected gradient descent (PGD) on the constraint $ S(boldsymbol)leq 2/$.<n>For mini-batch gradient descent
  arXiv  Detail & Related papers  (2026-04-22T19:02:52Z)
• Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks [74.46751089984072]
  gradient descent (SGD) noise on descent in deep linear networks is poorly understood.<n>We investigate the dynamics of SGD during training of DLNs in the saddle-to-saddle regime.<n>Our results establish that SGD noise encodes information about the progression of feature learning but does not fundamentally alter the saddle-to-saddle dynamics.
  arXiv  Detail & Related papers  (2026-04-07T18:43:08Z)
• A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation [17.31702538576744]
  We study the dynamics of gradient flow with small weight decay on general training losses $F: mathbbRd to mathbbR$.
  arXiv  Detail & Related papers  (2025-05-26T16:12:45Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used.<n>We provide a proof of this in the case of linear neural networks with a squared loss.<n>We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction [49.66486092259376]
  The mean-field Langevin dynamics (MFLD) is a nonlinear generalization of the Langevin dynamics that incorporates a distribution-dependent drift. Recent works have shown that MFLD globally minimizes an entropy-regularized convex functional in the space of measures. We provide a framework to prove a uniform-in-time propagation of chaos for MFLD that takes into account the errors due to finite-particle approximation, time-discretization, and gradient approximation.
  arXiv  Detail & Related papers  (2023-06-12T16:28:11Z)
• Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability [40.17821914923602]
  We show that gradient descent at the edge of stability implicitly follows projected gradient descent (PGD) under the constraint $S(theta) le 2/eta$. Our analysis provides precise predictions for the loss, sharpness, and deviation from the PGD trajectory throughout training.
  arXiv  Detail & Related papers  (2022-09-30T17:15:12Z)
• Losing momentum in continuous-time stochastic optimisation [42.617042045455506]
  momentum-based optimisation algorithms have become particularly widespread. In this work, we analyse a continuous-time model for gradient descent with momentum. We also train a convolutional neural network in an image classification problem.
  arXiv  Detail & Related papers  (2022-09-08T10:46:05Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Long-lived period-doubled edge modes of interacting and disorder-free Floquet spin chains [68.8204255655161]
  We show that even in the absence of disorder, and in the presence of bulk heating, $pi$ edge modes are long lived. A tunneling estimate for the lifetime is obtained by mapping the stroboscopic time-evolution to dynamics of a single particle in Krylov subspace.
  arXiv  Detail & Related papers  (2021-05-28T12:13:14Z)
• Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent [55.85456985750134]
  We introduce a new stability measure called on-average model stability, for which we develop novel bounds controlled by the risks of SGD iterates. This yields generalization bounds depending on the behavior of the best model, and leads to the first-ever-known fast bounds in the low-noise setting. To our best knowledge, this gives the firstever-known stability and generalization for SGD with even non-differentiable loss functions.
  arXiv  Detail & Related papers  (2020-06-15T06:30:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.