Momentum Further Constrains Sharpness at the Edge of Stochastic Stability

• URL: http://arxiv.org/abs/2604.14108v1
• Date: Wed, 15 Apr 2026 17:28:46 GMT
• Title: Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
• Abstract summary: Recent work suggests (stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found.<n>We demonstrate that SGD with momentum exhibits an Edge of Stability (EoSS)-like regime with batch-size-dependent behavior.
• Score: 0.5405981353784005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work suggests that (stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found. Momentum and mini-batch gradients are widely used in practical deep learning optimization, but it remains unclear whether they operate in a comparable regime of instability. We demonstrate that SGD with momentum exhibits an Edge of Stochastic Stability (EoSS)-like regime with batch-size-dependent behavior that cannot be explained by a single momentum-adjusted stability threshold. Batch Sharpness (the expected directional mini-batch curvature) stabilizes in two distinct regimes: at small batch sizes it converges to a lower plateau $2(1-β)/η$, reflecting amplification of stochastic fluctuations by momentum and favoring flatter regions than vanilla SGD; at large batch sizes it converges to a higher plateau $2(1+β)/η$, where momentum recovers its classical stabilizing effect and favors sharper regions consistent with full-batch dynamics. We further show that this aligns with linear stability thresholds and discuss the implications for hyperparameter tuning and coupling.

Related papers

• Variational Structure at the Edge of Stability [0.0]
  We show that the edge coupling can be identified with the symmetric Verlet action.<n>We also show that the edge coupling can be identified with the symmetric Verlet action.
  arXiv  Detail & Related papers  (2026-08-21T22:05:12Z)
• The Road Taken: The Role of Optimizers at the Edge of Stability [53.81268610971847]
  edge of stability refers to a phenomenon in deep learning where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent predicts to be unstable.<n>Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate.<n>We observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as $times 21.1$.
  arXiv  Detail & Related papers  (2026-08-19T01:03:34Z)
• SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates [2.104588729923885]
  Modern deep learning has been shown to operate at the edge of stability.<n>We provide convergence guarantees for Gradient Descent (SGD) applied to the multiclass cross-entropy loss.<n> Experiments validate our theoretical findings and illustrate the benefits of SGD in the large-stepsize regime.
  arXiv  Detail & Related papers  (2026-06-29T21:32:58Z)
• Distribution-Aware Robust Bilevel Optimization: Quantile-Guided Huber Updates in Two-Timescale Stochastic Approximation [16.106678611834102]
  Bilevel optimization (BLO) is fundamental to hierarchical decision-making but suffers from critical instability under heavy-tailed noise.<n>Existing techniques typically rely on magnitude checks, which fail to distinguish informative geometric signals from outliers.
  arXiv  Detail & Related papers  (2026-06-21T10:57:37Z)
• Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models [56.67321805551389]
  Reinforcement learning (RL) has become an effective way to improve prompt alignment and perceptual quality in diffusion and flow-matching generators.<n>The sampler, controlling the exploration behavior and denoising dynamics, is thus part of the policy.<n>We propose a new sampler that balances effective exploration with stability.
  arXiv  Detail & Related papers  (2026-05-22T11:37:22Z)
• Zeroth-Order Optimization at the Edge of Stability [59.54782674222443]
  We provide an explicit step size condition that captures the (mean-square) linear stability of a family of Zeroth-order (ZO) methods.<n>Our results highlight an implicit regularization effect specific to ZO methods, where large step sizes primarily regularize the Hessian trace.
  arXiv  Detail & Related papers  (2026-04-16T06:23:18Z)
• On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization [0.0]
  We analyze the tracking performance of Gradient Descent under uniform strong convexity and smoothness in varying stepsize regimes.<n>We show that momentum can substantially amplify drift-induced tracking error, with an explicit penalty on the tracking capability.<n>These results provide a definitive theoretical grounding for the empirical instability of momentum in dynamic environments.
  arXiv  Detail & Related papers  (2026-01-18T03:27:21Z)
• Breaking the Heavy-Tailed Noise Barrier in Stochastic Optimization Problems [56.86067111855056]
  We consider clipped optimization problems with heavy-tailed noise with structured density. We show that it is possible to get faster rates of convergence than $mathcalO(K-(alpha - 1)/alpha)$, when the gradients have finite moments of order. We prove that the resulting estimates have negligible bias and controllable variance.
  arXiv  Detail & Related papers  (2023-11-07T17:39:17Z)
• Exact Mean Square Linear Stability Analysis for SGD [28.65663421598186]
  We provide an explicit condition on the step size that is both necessary and sufficient for linear stability of gradient descent (SGD) We show that SGD's stability threshold is equivalent to that of a mixture process which takes in each a full batch gradient step w.p. $1-p$, and a single sample gradient step w.p. $p$.
  arXiv  Detail & Related papers  (2023-06-13T15:29:23Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentum [14.158845925610438]
  We show how a gradient proximal algorithm with momentum (PPA) allows faster convergence to a neighborhood compared to the proximal algorithm (PPA) with better contraction factor.
  arXiv  Detail & Related papers  (2021-11-11T12:17:22Z)
• On the Convergence of Stochastic Extragradient for Bilinear Games with Restarted Iteration Averaging [96.13485146617322]
  We present an analysis of the ExtraGradient (SEG) method with constant step size, and present variations of the method that yield favorable convergence. We prove that when augmented with averaging, SEG provably converges to the Nash equilibrium, and such a rate is provably accelerated by incorporating a scheduled restarting procedure.
  arXiv  Detail & Related papers  (2021-06-30T17:51:36Z)
• Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent [55.85456985750134]
  We introduce a new stability measure called on-average model stability, for which we develop novel bounds controlled by the risks of SGD iterates. This yields generalization bounds depending on the behavior of the best model, and leads to the first-ever-known fast bounds in the low-noise setting. To our best knowledge, this gives the firstever-known stability and generalization for SGD with even non-differentiable loss functions.
  arXiv  Detail & Related papers  (2020-06-15T06:30:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.