Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate

• URL: http://arxiv.org/abs/2604.08742v1
• Date: Thu, 09 Apr 2026 20:09:24 GMT
• Title: Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
• Abstract summary: Adam has achieved strong empirical success, but its theory remains incomplete even in the deterministic full-batch setting.<n>In this work, a convergent reformulation of full-batch Adam is developed by combining variable and operator splitting with a curvature-aware gradient correction.<n>This leads to a continuous-time Adam-HNAG flow with an exponentially decaying Lyapunov function, as well as two discrete methods: Adam-HNAG, and Adam-HNAG-s, a synchronous variant closer in form to Adam.
• Score: 7.396660927040262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adam has achieved strong empirical success, but its theory remains incomplete even in the deterministic full-batch setting, largely because adaptive preconditioning and momentum are tightly coupled. In this work, a convergent reformulation of full-batch Adam is developed by combining variable and operator splitting with a curvature-aware gradient correction. This leads to a continuous-time Adam-HNAG flow with an exponentially decaying Lyapunov function, as well as two discrete methods: Adam-HNAG, and Adam-HNAG-s, a synchronous variant closer in form to Adam. Within a unified Lyapunov analysis framework, convergence guarantees are established for both methods in the convex smooth setting, including accelerated convergence. Numerical experiments support the theory and illustrate the different empirical behavior of the two discretizations. To the best of our knowledge, this provides the first convergence proof for Adam-type methods in convex optimization.

Related papers

• Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization [6.759434764396595]
  We propose a Lyapunov-guided Adam-type method that couples momentum, adaptive preconditioning, and a curvature-conditioner correction.<n>Experiments validate the predicted decay and show competitive training performance against Adam and AdamW on deep learning tasks.
  arXiv  Detail & Related papers  (2026-05-13T01:46:23Z)
• Towards Understanding Adam Convergence on Highly Degenerate Polynomials [12.224244942795695]
  We study the "natural" auto-convergence properties of Adam.<n>We find a class of highly degenerates where Adam converges automatically without additional schedulers.<n>We prove that Adam degenerates local linear convergence on these functions, significantly outperforming the sub- convergence of Gradient Descent and Momentum.
  arXiv  Detail & Related papers  (2026-03-10T12:30:20Z)
• Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer [1.9015962471041752]
  We prove for a class of simple quadratic strongly convex SOPs that Adam converges as the number of gradient steps increases to infinity.<n>In particular, in the standard case where the random variables in the SOP are not symmetrically distributed we emphdisprove that Adam converges to the minimizer of the SOP as the number of Adam steps increases to infinity.
  arXiv  Detail & Related papers  (2025-11-10T03:45:32Z)
• A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD [28.905886549938305]
  We introduce a novel and comprehensive framework for analyzing the convergence properties of Adam.<n>We show that Adam attains non-asymptotic complexity sample bounds similar to those of gradient descent.
  arXiv  Detail & Related papers  (2024-10-06T12:15:00Z)
• On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond [35.65852208995095]
  We demonstrate that Adam achieves a faster convergence compared to SGDM under the condition of non-uniformly bounded smoothness. Our findings reveal that: (1) in deterministic environments, Adam can attain the known lower bound for the convergence rate of deterministic first-orders, whereas the convergence rate of Gradient Descent with Momentum (GDM) has higher order dependence on the initial function value.
  arXiv  Detail & Related papers  (2024-03-22T11:57:51Z)
• Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity [51.96093077151991]
  We derive a new convergence guarantee of Adam, with only an $L$-smooth condition and a bounded noise variance assumption. Our proof utilizes novel techniques to handle the entanglement between momentum and adaptive learning rate.
  arXiv  Detail & Related papers  (2023-10-27T09:16:58Z)
• Convergence of Adam Under Relaxed Assumptions [72.24779199744954]
  We show that Adam converges to $epsilon$-stationary points with $O(epsilon-4)$ gradient complexity under far more realistic conditions. We also propose a variance-reduced version of Adam with an accelerated gradient complexity of $O(epsilon-3)$.
  arXiv  Detail & Related papers  (2023-04-27T06:27:37Z)
• Provable Adaptivity of Adam under Non-uniform Smoothness [79.25087082434975]
  Adam is widely adopted in practical applications due to its fast convergence. Existing convergence analyses for Adam rely on the bounded smoothness assumption. This paper studies the convergence of randomly reshuffled Adam with diminishing learning rate.
  arXiv  Detail & Related papers  (2022-08-21T14:57:47Z)
• A Novel Convergence Analysis for Algorithms of the Adam Family [105.22760323075008]
  We present a generic proof of convergence for a family of Adam-style methods including Adam, AMSGrad, Adabound, etc. Our analysis is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non- compositional optimization problems.
  arXiv  Detail & Related papers  (2021-12-07T02:47:58Z)
• Adam$^+$: A Stochastic Method with Adaptive Variance Reduction [56.051001950733315]
  Adam is a widely used optimization method for deep learning applications. We propose a new method named Adam$+$ (pronounced as Adam-plus) Our empirical studies on various deep learning tasks, including image classification, language modeling, and automatic speech recognition, demonstrate that Adam$+$ significantly outperforms Adam.
  arXiv  Detail & Related papers  (2020-11-24T09:28:53Z)
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients [112.00379151834242]
  We propose adaptive learning rate principle, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance each coordinate. This results in faster adaptation, which leads more desirable empirical convergence behaviors.
  arXiv  Detail & Related papers  (2020-06-21T21:47:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.