From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamics

• URL: http://arxiv.org/abs/2602.09101v1
• Date: Mon, 09 Feb 2026 19:00:03 GMT
• Title: From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamics
• Abstract summary: We derive an accelerated continuous-time formulation of Adam by modeling it as a second-order integro-differential dynamical system.<n>We relate this inertial nonlocal model to an existing first-order nonlocal Adam flow through an $$-refinement limit, and we provide Lyapunov-based stability and convergence analyses.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we derive an accelerated continuous-time formulation of Adam by modeling it as a second-order integro-differential dynamical system. We relate this inertial nonlocal model to an existing first-order nonlocal Adam flow through an $α$-refinement limit, and we provide Lyapunov-based stability and convergence analyses. We also introduce an Adam-inspired nonlocal Lagrangian formulation, offering a variational viewpoint. Numerical simulations on Rosenbrock-type examples show agreement between the proposed dynamics and discrete Adam.

Related papers

• Provable Edge-of-Stability for Adam on a One-Dimensional Quadratic [4.5845031477639155]
  We study uncorrected Adam on a one-dimensional quadratic, a clean setting where constant curvature isolates the edge-induced dynamics behind the EoS.<n>We identify settings in which this edge-seeking mechanism breaks down, including strictly subcritical periodic orbits and specially tuned trajectories that converge to the optimum.<n>These results give a concrete dynamical explanation for Adam's EoS in a setting free of evolving loss geometry, while also exposing its limitations.
  arXiv  Detail & Related papers  (2026-08-21T00:22:11Z)
• Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics [92.39053980710702]
  Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion.<n>This approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate.<n>We propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:40Z)
• Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach [14.605735105736864]
  We derive ordinary differential equations that serve as continuous-time limits of the Adam-DA.<n>Our analysis reveals that the roles of the first- and second-order momentum parameters in zero-sum games are exactly the opposite of their well-documented effects in minimization problems.
  arXiv  Detail & Related papers  (2026-05-19T05:38:47Z)
• Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate [7.396660927040262]
  Adam has achieved strong empirical success, but its theory remains incomplete even in the deterministic full-batch setting.<n>In this work, a convergent reformulation of full-batch Adam is developed by combining variable and operator splitting with a curvature-aware gradient correction.<n>This leads to a continuous-time Adam-HNAG flow with an exponentially decaying Lyapunov function, as well as two discrete methods: Adam-HNAG, and Adam-HNAG-s, a synchronous variant closer in form to Adam.
  arXiv  Detail & Related papers  (2026-04-09T20:09:24Z)
• Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails [66.18297682243694]
  We uncover a key second-moment normalization in Adam and develop a stopping-time/martingale analysis that distinguishes Adam from SGD.<n>In particular, we establish the first theoretical separation between the high-probability convergence behaviors of the two methods.
  arXiv  Detail & Related papers  (2026-03-03T15:34:51Z)
• Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling [36.106114687828395]
  Adam is known to perform significantly better than Gradient Descent (SGD) in language models.<n>We exhaustively study how momentum, gradient clipping, and batch size affect the gap between SGD and Adam.
  arXiv  Detail & Related papers  (2025-06-14T15:37:31Z)
• In Search of Adam's Secret Sauce [11.215133680044005]
  We train over 1,300 language models across different data configurations and scales.<n>We find that signed momentum methods are faster than SGD, but consistently underperform relative to Adam.<n>We show that Adam in this setting implements a natural online algorithm for estimating the mean and variance of gradients.
  arXiv  Detail & Related papers  (2025-05-27T23:30:18Z)
• Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization [90.08459757321405]
  Federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead.<n>We propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates local model parameters and moment estimates.<n>By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error.
  arXiv  Detail & Related papers  (2024-05-28T07:56:49Z)
• Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity [51.96093077151991]
  We derive a new convergence guarantee of Adam, with only an $L$-smooth condition and a bounded noise variance assumption. Our proof utilizes novel techniques to handle the entanglement between momentum and adaptive learning rate.
  arXiv  Detail & Related papers  (2023-10-27T09:16:58Z)
• Adam-family Methods with Decoupled Weight Decay in Deep Learning [3.4376560669160394]
  We investigate the convergence properties of a wide of Adam-family methods for nonsmooth nonsmooth networks. We propose a novel Adam-family method named Adam with Decoupled Weight Decay (AdamD) in our proposed framework.
  arXiv  Detail & Related papers  (2023-10-13T04:59:44Z)
• UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization [20.399244578926474]
  We introduce a unified framework for Adam-type algorithms (called UAdam) This is equipped with a general form of the second-order moment, such as NAdamBound, AdaFom, and Adan. We show that UAdam converges to the neighborhood of stationary points with the rate of $mathcalO (1/T)$.
  arXiv  Detail & Related papers  (2023-05-09T13:07:03Z)
• Convergence of Adam Under Relaxed Assumptions [72.24779199744954]
  We show that Adam converges to $epsilon$-stationary points with $O(epsilon-4)$ gradient complexity under far more realistic conditions. We also propose a variance-reduced version of Adam with an accelerated gradient complexity of $O(epsilon-3)$.
  arXiv  Detail & Related papers  (2023-04-27T06:27:37Z)
• Provable Adaptivity of Adam under Non-uniform Smoothness [79.25087082434975]
  Adam is widely adopted in practical applications due to its fast convergence. Existing convergence analyses for Adam rely on the bounded smoothness assumption. This paper studies the convergence of randomly reshuffled Adam with diminishing learning rate.
  arXiv  Detail & Related papers  (2022-08-21T14:57:47Z)
• A Novel Convergence Analysis for Algorithms of the Adam Family [105.22760323075008]
  We present a generic proof of convergence for a family of Adam-style methods including Adam, AMSGrad, Adabound, etc. Our analysis is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non- compositional optimization problems.
  arXiv  Detail & Related papers  (2021-12-07T02:47:58Z)
• Adam$^+$: A Stochastic Method with Adaptive Variance Reduction [56.051001950733315]
  Adam is a widely used optimization method for deep learning applications. We propose a new method named Adam$+$ (pronounced as Adam-plus) Our empirical studies on various deep learning tasks, including image classification, language modeling, and automatic speech recognition, demonstrate that Adam$+$ significantly outperforms Adam.
  arXiv  Detail & Related papers  (2020-11-24T09:28:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.