Scaling Limits of Constant-Stepsize SGD at Flat Minima

• URL: http://arxiv.org/abs/2607.16384v1
• Date: Fri, 17 Jul 2026 17:22:46 GMT
• Title: Scaling Limits of Constant-Stepsize SGD at Flat Minima
• Abstract summary: We study Markovian noise generated by a contractive driving chain.<n>For gradient descent (SGD) with a constant stepsize $$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons.<n>We show that this behavior changes fundamentally for convex objectives $H$ with flat minima and (sub)quadratic tails.
• Score: 5.608222858044445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For stochastic gradient descent (SGD) with a constant stepsize $α$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar $\sqrtα$ scaling and a Gaussian limit as $α\downarrow 0$. We show that this behavior changes fundamentally for convex objectives $H$ with flat minima and (sub)quadratic tails. More specifically, we study SGD with Markovian noise generated by a contractive driving chain. For every sufficiently small constant stepsize $α$, we prove existence, uniqueness, and geometric convergence to an augmented invariant law in a Wasserstein distance induced by an $α$-dependent metric. When the minimizer $x_\star$ has local flatness exponent $m\ge2$, meaning that $\nabla^2 H(x)\asymp \lVert x-x_\star\rVert^{m-2} I_d$ as $x\to x_\star$, we obtain a contraction bound with factor $1-cα^{m-1}$, where $c>0$ is a constant. This recovers the factor $1-cα$ in the quadratic case $m=2$. We then analyze the small-stepsize scaling limit. We show that the invariant law concentrates on the scale $α^{1/m}$ and that the rescaled iterates converge weakly to the stationary distribution of the stochastic differential equation $$ dY_t=-h_0(Y_t)\,dt+Σ^{1/2}\,dB_t , $$ where $h_0$ is the limiting drift at the minimizer and $Σ$ denotes the asymptotic covariance. This recovers the Gaussian limit when $m=2$ and gives generally non-Gaussian stationary limits in the flat case $m>2$. Finally, we give corresponding results for coordinate-separable objectives with unequal flatness exponents.

Related papers

• One Inverse Step is a Convex Program: Bayes-Limit Calibration of Diffusion Inversion [0.0]
  One implicit DDIM inversion step is the cheapest probe of whether a pretrained diffusion model encodes local manifold geometry.<n>The Fermi window conflicts with the model's own training support by $3.6$-$5.6times$, and the trained Hessian-Lipschitz constant is $2$-$12%$ of the curvature the law reads.<n>Finally the unconditional ceiling $_t_max(mathrmsym,J)le1$, from $mathrmCov(x_0mid x_t)succeq0$
  arXiv  Detail & Related papers  (2026-08-24T11:01:54Z)
• High Minima of Gaussian Processes: Overshoots and Minimizer Locations [0.0]
  We show that, conditionally on $M>u$, the scaled overshoot $u(M-u)$ converges, as $utoinfty$, to an exponential random variable with mean $_*2$.<n>The results are illustrated by stationary Gaussian processes, fractional Brownian motion, and fractional Brownian sheet.
  arXiv  Detail & Related papers  (2026-07-22T20:31:36Z)
• Asymptotically Optimal Regret for Reinforcement Learning without Horizon Dependence [54.59847568544922]
  We study horizon-free regret for finite-horizon time-homogeneous Markov decision processes with $A$ states, $A$ actions, horizon $H$, and per-trajectory total reward bounded by $1$.<n>We prove a regret upper bound [tilde O(sqrtSAK+S8A3)] with failure probability $$, where $K$ is the number of episodes and $tilde O(sqrtSAK+S3K)$ hides $mathsfpoly
  arXiv  Detail & Related papers  (2026-07-22T07:42:19Z)
• Solving Stochastic Fixed-Point Equations with High Probability [22.376855234542813]
  We study oracle fixed-point equations $mathbfT(mathbfx) = mathbfx$ over normed spaces.<n>We introduce VR-GHAL, a variance-reduced gradual Halpern method for quadratically smoothable Banach spaces.
  arXiv  Detail & Related papers  (2026-07-10T04:59:20Z)
• Estimation of the sub-Gaussian parameter [7.4076878426925035]
  The sub-Gaussian random variable $X$ is defined as $2_* = s_upin mathbbR L()$ where $L() = frac22 log mathbbE eX$ is a weighted cumulant generating function.<n>Despite the ubiquity of sub-Gaussian random variables, the estimation of $2_*$ has received little attention and is not yet well understood.
  arXiv  Detail & Related papers  (2026-06-04T16:48:31Z)
• When Does $\ell_2$-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the $\ell_1$ Implicit Bias [15.113649527486276]
  We show that benign overfitting fails at the linear rate.<n>We show that while this localization mechanism should persist in the presence of signals, the exact signal-noise decomposition remains an open problem.
  arXiv  Detail & Related papers  (2026-05-07T14:14:09Z)
• Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy [0.0]
  Cross-entropy training analyses rely on local Taylor models of the loss to predict whether a proposed step will decrease the objective.<n>We derive closed-form expressions under logit linearization along the proposed update direction.<n>Temperature scaling confirms the mechanism: normalizing by $_a$ shrinks the onset-threshold spread from standard deviation $0.992$ to $0.164$.
  arXiv  Detail & Related papers  (2026-03-13T19:42:12Z)
• Near-Optimal Convergence of Accelerated Gradient Methods under Generalized and $(L_0, L_1)$-Smoothness [57.93371273485736]
  We study first-order methods for convex optimization problems with functions $f$ satisfying the recently proposed $ell$-smoothness condition $||nabla2f(x)|| le ellleft(||nabla f(x)||right),$ which generalizes the $L$-smoothness and $(L_0,L_1)$-smoothness.
  arXiv  Detail & Related papers  (2025-08-09T08:28:06Z)
• On the $O(\frac{\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1$ Norm [54.28350823319057]
  This paper considers the RMSProp and its momentum extension and establishes the convergence rate of $frac1Tsum_k=1T.<n>Our convergence rate matches the lower bound with respect to all the coefficients except the dimension $d$.<n>Our convergence rate can be considered to be analogous to the $frac1Tsum_k=1T.
  arXiv  Detail & Related papers  (2024-02-01T07:21:32Z)
• Measurement-induced phase transition for free fermions above one dimension [46.176861415532095]
  Theory of the measurement-induced entanglement phase transition for free-fermion models in $d>1$ dimensions is developed. Critical point separates a gapless phase with $elld-1 ln ell$ scaling of the second cumulant of the particle number and of the entanglement entropy.
  arXiv  Detail & Related papers  (2023-09-21T18:11:04Z)
• A first-order primal-dual method with adaptivity to local smoothness [64.62056765216386]
  We consider the problem of finding a saddle point for the convex-concave objective $min_x max_y f(x) + langle Ax, yrangle - g*(y)$, where $f$ is a convex function with locally Lipschitz gradient and $g$ is convex and possibly non-smooth. We propose an adaptive version of the Condat-Vu algorithm, which alternates between primal gradient steps and dual steps.
  arXiv  Detail & Related papers  (2021-10-28T14:19:30Z)
• Private Stochastic Convex Optimization: Optimal Rates in $\ell_1$ Geometry [69.24618367447101]
  Up to logarithmic factors the optimal excess population loss of any $(varepsilon,delta)$-differently private is $sqrtlog(d)/n + sqrtd/varepsilon n.$ We show that when the loss functions satisfy additional smoothness assumptions, the excess loss is upper bounded (up to logarithmic factors) by $sqrtlog(d)/n + (log(d)/varepsilon n)2/3.
  arXiv  Detail & Related papers  (2021-03-02T06:53:44Z)
• A Simple Convergence Proof of Adam and Adagrad [74.24716715922759]
  We show a proof of convergence between the Adam Adagrad and $O(d(N)/st)$ algorithms. Adam converges with the same convergence $O(d(N)/st)$ when used with the default parameters.
  arXiv  Detail & Related papers  (2020-03-05T01:56:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.