Towards Understanding Adam Convergence on Highly Degenerate Polynomials

• URL: http://arxiv.org/abs/2603.09581v1
• Date: Tue, 10 Mar 2026 12:30:20 GMT
• Title: Towards Understanding Adam Convergence on Highly Degenerate Polynomials
• Abstract summary: We study the "natural" auto-convergence properties of Adam.<n>We find a class of highly degenerates where Adam converges automatically without additional schedulers.<n>We prove that Adam degenerates local linear convergence on these functions, significantly outperforming the sub- convergence of Gradient Descent and Momentum.
• Score: 12.224244942795695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $β_2$ near 1 for convergence, this work investigates the "natural" auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.

Related papers

• Convergence of Steepest Descent and Adam under Non-Uniform Smoothness [9.95160888054888]
  We show that for logistic regression on separable data, RMSProp and Adam converge linearly and is provably faster than GD.<n>We also show that for a class of two-layer neural networks on separable data, RMSProp and Adam can converge at a linear rate with a constant step-size and momentum parameter.
  arXiv  Detail & Related papers  (2026-05-28T23:05:45Z)
• Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer [1.9015962471041752]
  We prove for a class of simple quadratic strongly convex SOPs that Adam converges as the number of gradient steps increases to infinity.<n>In particular, in the standard case where the random variables in the SOP are not symmetrically distributed we emphdisprove that Adam converges to the minimizer of the SOP as the number of Adam steps increases to infinity.
  arXiv  Detail & Related papers  (2025-11-10T03:45:32Z)
• High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noise [4.9495085874952895]
  We show that Adam could converge to the stationary point in high probability with a rate of $mathcalOleft(rm poly(log T)/sqrtTright)$ under coordinate-wise "affine" noise variance. It is also revealed that Adam's confines within an order of $mathcalOleft(rm poly(left T)right)$ are adaptive to the noise level.
  arXiv  Detail & Related papers  (2023-11-03T15:55:53Z)
• Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity [51.96093077151991]
  We derive a new convergence guarantee of Adam, with only an $L$-smooth condition and a bounded noise variance assumption. Our proof utilizes novel techniques to handle the entanglement between momentum and adaptive learning rate.
  arXiv  Detail & Related papers  (2023-10-27T09:16:58Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Convergence of Adam Under Relaxed Assumptions [72.24779199744954]
  We show that Adam converges to $epsilon$-stationary points with $O(epsilon-4)$ gradient complexity under far more realistic conditions. We also propose a variance-reduced version of Adam with an accelerated gradient complexity of $O(epsilon-3)$.
  arXiv  Detail & Related papers  (2023-04-27T06:27:37Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Convex Analysis of the Mean Field Langevin Dynamics [49.66486092259375]
  convergence rate analysis of the mean field Langevin dynamics is presented. $p_q$ associated with the dynamics allows us to develop a convergence theory parallel to classical results in convex optimization.
  arXiv  Detail & Related papers  (2022-01-25T17:13:56Z)
• A Novel Convergence Analysis for Algorithms of the Adam Family [105.22760323075008]
  We present a generic proof of convergence for a family of Adam-style methods including Adam, AMSGrad, Adabound, etc. Our analysis is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non- compositional optimization problems.
  arXiv  Detail & Related papers  (2021-12-07T02:47:58Z)
• A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms [12.865834066050427]
  RMSprop and Adam algorithms are studied through a combination of careful numerical experiments and theoretical explanations. Three types of qualitative features are observed in the training loss curve: fast initial convergence, oscillations, and large spikes in the late phase.
  arXiv  Detail & Related papers  (2020-09-14T00:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.