The Origin of Edge of Stability

• URL: http://arxiv.org/abs/2604.20446v1
• Date: Wed, 22 Apr 2026 11:08:49 GMT
• Title: The Origin of Edge of Stability
• Abstract summary: Full-batch gradient descent on neural networks drives the largest Hessian eigenvalue to the threshold $2/$, where $$ is the learning rate.<n>We introduce the edge coupling, a functional on consecutive iterate pairs whose coefficient is uniquely fixed by the gradient-descent update.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Full-batch gradient descent on neural networks drives the largest Hessian eigenvalue to the threshold $2/η$, where $η$ is the learning rate. This phenomenon, the Edge of Stability, has resisted a unified explanation: existing accounts establish self-regulation near the edge but do not explain why the trajectory is forced toward $2/η$ from arbitrary initialization. We introduce the edge coupling, a functional on consecutive iterate pairs whose coefficient is uniquely fixed by the gradient-descent update. Differencing its criticality condition yields a step recurrence with stability boundary $2/η$, and a second-order expansion yields a loss-change formula whose telescoping sum forces curvature toward $2/η$. The two formulas involve different Hessian averages, but the mean value theorem localizes each to the true Hessian at an interior point of the step segment, yielding exact forcing of the Hessian eigenvalue with no gap. Setting both gradients of the edge coupling to zero classifies fixed points and period-two orbits; near a fixed point, the problem reduces to a function of the half-amplitude alone, which determines which directions support period-two orbits and on which side of the critical learning rate they appear.

Related papers

• Constrained Online Learning with Noisy Constraint Values [55.29259818039367]
  Under common feasibility, our LEDGER algorithm attains $O(sqrt T)$ expected regret and $O(sqrtTlog(eT))$ expected budget violation.<n>Neither a Slater condition, independence between feedback channels, nor an absolute constraint-value bound is needed.
  arXiv  Detail & Related papers  (2026-09-07T01:38:41Z)
• Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks [16.84694273405234]
  We study the identity straight-through estimator (STE) for training a two-layer binary-activation network with hinge loss.<n>Our central question is whether algorithmic stability can explain the statistical generalization of the estimator produced by the discontinuous STE training rule.
  arXiv  Detail & Related papers  (2026-09-06T07:12:16Z)
• Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data [3.7437735326653883]
  This paper studies the rate-controlled middle case for full-batch linear classification on separable data.<n>For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice.
  arXiv  Detail & Related papers  (2026-07-07T08:55:35Z)
• Sample Complexity of Policy Gradient for Log-Growth Control [23.86784769848408]
  We study the sample complexity of policy gradient for log-growth control.<n>We prove that projected mini-batch policy gradient attains total sample complexity.
  arXiv  Detail & Related papers  (2026-05-26T07:24:14Z)
• SGD at the Edge of Stability: The Stochastic Sharpness Gap [10.176501817419371]
  When training networks with full-batch gradient descent (GD) and step size $$, the largest eigenvalue of the Hessian rises to $2/$ and hovers there, a phenomenon termed the Edge of Stability (EoS)<n>citetdamian2023selfstab showed that this behavior is explained by a self-stabilization mechanism driven by third-order structure of the loss, and that GD implicitly follows projected gradient descent (PGD) on the constraint $ S(boldsymbol)leq 2/$.<n>For mini-batch gradient descent
  arXiv  Detail & Related papers  (2026-04-22T19:02:52Z)
• Conservation Law Breaking at the Edge of Stability: A Spectral Theory of Non-Convex Neural Network Optimization [0.0]
  We show that gradient descent reliably finds good solutions in non-dimensional network optimization.<n>We identify two regimes separated by a width-dependent dynamical transition.<n>All predictions are validated across 23 experiments.
  arXiv  Detail & Related papers  (2026-04-08T10:41:24Z)
• Improving Online-to-Nonconvex Conversion for Smooth Optimization via Double Optimism [25.642618010943824]
  We introduce an online optimistic gradient method based on a novel doubly optimistic hint function.<n>We obtain an algorithm unified with $1.75 + sigma2 varepsilon-3.5)$, smoothly interpolating between the best-known deterministic rate and the optimal rate.
  arXiv  Detail & Related papers  (2025-10-03T16:41:24Z)
• Tunable quantum criticality and pseudocriticality across the fixed-point annihilation in the anisotropic spin-boson model [0.26107298043931204]
  We study the nontrivial renormalization-group scenario of fixed-point annihilation in spin-boson models. We find a tunable transition between two localized phases that can be continuous or strongly first-order. We also find scaling behavior at the symmetry-enhanced first-order transition, for which the inverse correlation-length exponent is given by the bath exponent.
  arXiv  Detail & Related papers  (2024-03-04T19:00:07Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used.<n>We provide a proof of this in the case of linear neural networks with a squared loss.<n>We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron [49.45105570960104]
  We prove the global convergence of randomly gradient descent with a $Oleft(T-3right)$ rate. These two bounds jointly give an exact characterization of the convergence rate. We show this potential function converges slowly, which implies the slow convergence rate of the loss function.
  arXiv  Detail & Related papers  (2023-02-20T15:33:26Z)
• Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability [40.17821914923602]
  We show that gradient descent at the edge of stability implicitly follows projected gradient descent (PGD) under the constraint $S(theta) le 2/eta$. Our analysis provides precise predictions for the loss, sharpness, and deviation from the PGD trajectory throughout training.
  arXiv  Detail & Related papers  (2022-09-30T17:15:12Z)
• Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimization [94.19177623349947]
  Non-smooth non optimization problems emerge in machine learning and business making. Two core challenges impede the development of efficient methods with finitetime convergence guarantee. Two-phase versions of GFM and SGFM are also proposed and proven to achieve improved large-deviation results.
  arXiv  Detail & Related papers  (2022-09-12T06:53:24Z)
• Optimal Extragradient-Based Bilinearly-Coupled Saddle-Point Optimization [116.89941263390769]
  We consider the smooth convex-concave bilinearly-coupled saddle-point problem, $min_mathbfxmax_mathbfyF(mathbfx) + H(mathbfx,mathbfy)$, where one has access to first-order oracles for $F$, $G$ as well as the bilinear coupling function $H$. We present a emphaccelerated gradient-extragradient (AG-EG) descent-ascent algorithm that combines extragrad
  arXiv  Detail & Related papers  (2022-06-17T06:10:20Z)
• Superdiffusion in random two dimensional system with time-reversal symmetry and long-range hopping [45.873301228345696]
  localization problem in the crossover regime for the dimension $d=2$ and hopping $V(r) propto r-2$ is not resolved yet. We show that for the hopping determined by two-dimensional anisotropic dipole-dipole interactions there exist two distinguishable phases at weak and strong disorder.
  arXiv  Detail & Related papers  (2022-05-29T16:53:20Z)
• High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails [55.561406656549686]
  We consider non- Hilbert optimization using first-order algorithms for which the gradient estimates may have tails. We show that a combination of gradient, momentum, and normalized gradient descent convergence to critical points in high-probability with best-known iteration for smooth losses.
  arXiv  Detail & Related papers  (2021-06-28T00:17:01Z)
• Anisotropy-mediated reentrant localization [62.997667081978825]
  We consider a 2d dipolar system, $d=2$, with the generalized dipole-dipole interaction $sim r-a$, and the power $a$ controlled experimentally in trapped-ion or Rydberg-atom systems. We show that the spatially homogeneous tilt $beta$ of the dipoles giving rise to the anisotropic dipole exchange leads to the non-trivial reentrant localization beyond the locator expansion.
  arXiv  Detail & Related papers  (2020-01-31T19:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.