HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization

• URL: http://arxiv.org/abs/2603.02649v1
• Date: Tue, 03 Mar 2026 06:29:24 GMT
• Title: HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization
• Authors: Feihu Huang, Guanyi Zhang, Songcan Chen,
• Abstract summary: We propose a class of efficient Adam (i.e., HomeAdam(W)) algorithms via sometimes returning momentum-based SGD.<n>We prove that our HomeAdam(W) have a smaller generalization error $O(frac1N)$ than $O(frachat-2TN)$ of Adam(W)-srf, since $hat$ is generally very small.
• Score: 43.39364515909059
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Adam and AdamW are a class of default optimizers for training deep learning models in machine learning. These adaptive algorithms converge faster but generalize worse compared to SGD. In fact, their proved generalization error $O(\frac{1}{\sqrt{N}})$ also is larger than $O(\frac{1}{N})$ of SGD, where $N$ denotes training sample size. Recently, although some variants of Adam have been proposed to improve its generalization, their improved generalizations are still unexplored in theory. To fill this gap, in the paper, we restudy generalization of Adam and AdamW via algorithmic stability, and first prove that Adam and AdamW without square-root (i.e., Adam(W)-srf) have a generalization error $O(\frac{\hatρ^{-2T}}{N})$, where $T$ denotes iteration number and $\hatρ>0$ denotes the smallest element of second-order momentum plus a small positive number. To improve generalization, we propose a class of efficient clever Adam (i.e., HomeAdam(W)) algorithms via sometimes returning momentum-based SGD. Moreover, we prove that our HomeAdam(W) have a smaller generalization error $O(\frac{1}{N})$ than $O(\frac{\hatρ^{-2T}}{N})$ of Adam(W)-srf, since $\hatρ$ is generally very small. In particular, it is also smaller than the existing $O(\frac{1}{\sqrt{N}})$ of Adam(W). Meanwhile, we prove our HomeAdam(W) have a faster convergence rate of $O(\frac{1}{T^{1/4}})$ than $O(\frac{\breveρ^{-1}}{T^{1/4}})$ of the Adam(W)-srf, where $\breveρ\leq\hatρ$ also is very small. Extensive numerical experiments demonstrate efficiency of our HomeAdam(W) algorithms.

Related papers

• Simple Convergence Proof of Adam From a Sign-like Descent Perspective [58.89890024903816]
  We show that Adam achieves the optimal rate of $cal O(frac1Ts14)$ rather than the previous $cal O(fracln TTs14)$.<n>Our theoretical analysis provides new insights into the role of momentum as a key factor ensuring convergence.
  arXiv  Detail & Related papers  (2025-07-08T13:19:26Z)
• Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity [6.270305440413688]
  We find that Adam performs worse when the favorable $ell_infty$-geometry is SGD while provably remains unaffected.<n>Our experiments confirm that Adam performs much worse when the favorable $ell_infty$-geometry is SGD while provably remains unaffected.
  arXiv  Detail & Related papers  (2024-10-10T17:58:53Z)
• Adam-mini: Use Fewer Learning Rates To Gain More [29.170425801678952]
  Adam-mini reduces memory cutting down the learning rate resources in Adam.<n>Adam-mini achieves on par or better performance than AdamW with 50% less memory footprint.
  arXiv  Detail & Related papers  (2024-06-24T16:56:41Z)
• Provable Adaptivity of Adam under Non-uniform Smoothness [79.25087082434975]
  Adam is widely adopted in practical applications due to its fast convergence. Existing convergence analyses for Adam rely on the bounded smoothness assumption. This paper studies the convergence of randomly reshuffled Adam with diminishing learning rate.
  arXiv  Detail & Related papers  (2022-08-21T14:57:47Z)
• Understanding AdamW through Proximal Methods and Scale-Freeness [57.47324825501137]
  Adam is a generalization of the $ell$ regularizer Adam-$ell$. AdamW decouples the gradient of Adam-$ell$ from the update rule of Adam-$ell$. We show that AdamW exhibits an advantage over Adam-$ell$ and the degree to which we expect the gradients of the network to exhibit multiple scales.
  arXiv  Detail & Related papers  (2022-01-31T21:00:55Z)
• Adam$^+$: A Stochastic Method with Adaptive Variance Reduction [56.051001950733315]
  Adam is a widely used optimization method for deep learning applications. We propose a new method named Adam$+$ (pronounced as Adam-plus) Our empirical studies on various deep learning tasks, including image classification, language modeling, and automatic speech recognition, demonstrate that Adam$+$ significantly outperforms Adam.
  arXiv  Detail & Related papers  (2020-11-24T09:28:53Z)
• A new regret analysis for Adam-type algorithms [78.825194932103]
  In theory, regret guarantees for online convex optimization require a rapidly decaying $beta_1to0$ schedule. We propose a novel framework that allows us to derive optimal, data-dependent regret bounds with a constant $beta_1$.
  arXiv  Detail & Related papers  (2020-03-21T19:19:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.