Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers

• URL: http://arxiv.org/abs/2603.07122v1
• Date: Sat, 07 Mar 2026 09:15:30 GMT
• Title: Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers
• Abstract summary: In the training of neural networks, adaptive moment estimation (Adam) typically converges fast but exhibits suboptimal generalization performance.<n>To enhance its ability to find flat minima, we propose its new variant named inverse Adam (InvAdam)<n>InvAdam computes element-wise multiplication of the first-order and second-order moments, while Adam computes the element-wise division of these two moments.
• Score: 57.049014152026864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the training of neural networks, adaptive moment estimation (Adam) typically converges fast but exhibits suboptimal generalization performance. A widely accepted explanation for its defect in generalization is that it often tends to converge to sharp minima. To enhance its ability to find flat minima, we propose its new variant named inverse Adam (InvAdam). The key improvement of InvAdam lies in its parameter update mechanism, which is opposite to that of Adam. Specifically, it computes element-wise multiplication of the first-order and second-order moments, while Adam computes the element-wise division of these two moments. This modification aims to increase the step size of the parameter update when the elements in the second-order moments are large and vice versa, which helps the parameter escape sharp minima and stay at flat ones. However, InvAdam's update mechanism may face challenges in convergence. To address this challenge, we propose dual Adam (DualAdam), which integrates the update mechanisms of both Adam and InvAdam, ensuring convergence while enhancing generalization performance. Additionally, we introduce the diffusion theory to mathematically demonstrate InvAdam's ability to escape sharp minima. Extensive experiments are conducted on image classification tasks and large language model (LLM) fine-tuning. The results validate that DualAdam outperforms Adam and its state-of-the-art variants in terms of generalization performance. The code is publicly available at https://github.com/LongJin-lab/DualAdam.

Related papers

• On the Principles Behind Neural Network Optimizers [4.4366291058606535]
  This thesis develops a principled grounding for Adam and motivates new principled structures.<n>We find that the Hessian structure evolves toward a near-block-diagonal preconditioner along training.<n>We prove that this structure makes Adam's diagonal preconditioner effective.
  arXiv  Detail & Related papers  (2026-08-17T16:07:25Z)
• MAdam: Metric-Aware Multi-Objective Adam [12.820085175292377]
  We show two systematic gaps between the solver's intent and the coupling's execution.<n>We introduce textbfMAdam (Metric-Aware Multi-Objective Adam), a drop-in wrapper that leaves both solver and unchanged.<n>MAdam consistently improves over Adam for every solver family.
  arXiv  Detail & Related papers  (2026-06-02T17:00:15Z)
• Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks [38.11287525994738]
  We present the first theoretical characterization of how affects Adam's generalization.<n>Our results reveal that while both Adam and AdamW with proper weight decay converge to poor test error solutions, their mini-batch variants can achieve near-zero test error.
  arXiv  Detail & Related papers  (2025-10-13T12:48:22Z)
• Provable Adaptivity of Adam under Non-uniform Smoothness [79.25087082434975]
  Adam is widely adopted in practical applications due to its fast convergence. Existing convergence analyses for Adam rely on the bounded smoothness assumption. This paper studies the convergence of randomly reshuffled Adam with diminishing learning rate.
  arXiv  Detail & Related papers  (2022-08-21T14:57:47Z)
• Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam [49.426602335460295]
  1-bit communication is an effective method to scale up model training, and has been studied extensively on SGD. We propose 0/1 Adam, which improves upon the state-of-the-art 1-bit Adam via two novel methods.
  arXiv  Detail & Related papers  (2022-02-12T08:02:23Z)
• Understanding AdamW through Proximal Methods and Scale-Freeness [57.47324825501137]
  Adam is a generalization of the $ell$ regularizer Adam-$ell$. AdamW decouples the gradient of Adam-$ell$ from the update rule of Adam-$ell$. We show that AdamW exhibits an advantage over Adam-$ell$ and the degree to which we expect the gradients of the network to exhibit multiple scales.
  arXiv  Detail & Related papers  (2022-01-31T21:00:55Z)
• A Novel Convergence Analysis for Algorithms of the Adam Family [105.22760323075008]
  We present a generic proof of convergence for a family of Adam-style methods including Adam, AMSGrad, Adabound, etc. Our analysis is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non- compositional optimization problems.
  arXiv  Detail & Related papers  (2021-12-07T02:47:58Z)
• Adam$^+$: A Stochastic Method with Adaptive Variance Reduction [56.051001950733315]
  Adam is a widely used optimization method for deep learning applications. We propose a new method named Adam$+$ (pronounced as Adam-plus) Our empirical studies on various deep learning tasks, including image classification, language modeling, and automatic speech recognition, demonstrate that Adam$+$ significantly outperforms Adam.
  arXiv  Detail & Related papers  (2020-11-24T09:28:53Z)
• EAdam Optimizer: How $\epsilon$ Impact Adam [7.0552555621312605]
  We discuss the impact of the constant $epsilon$ for Adam in this paper. Based on this finding, we propose a new variant of Adam called EAdam. Our method can bring significant improvement compared with Adam.
  arXiv  Detail & Related papers  (2020-11-04T06:39:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.