On the Principles Behind Neural Network Optimizers

• URL: http://arxiv.org/abs/2608.16760v1
• Date: Mon, 17 Aug 2026 16:07:25 GMT
• Title: On the Principles Behind Neural Network Optimizers
• Abstract summary: This thesis develops a principled grounding for Adam and motivates new principled structures.<n>We find that the Hessian structure evolves toward a near-block-diagonal preconditioner along training.<n>We prove that this structure makes Adam's diagonal preconditioner effective.
• Score: 4.4366291058606535
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation. This thesis develops a principled grounding for Adam and motivates new designs. First, we revisit Adam's divergence--convergence debate and show the existence of a problem-dependent phase transition: with properly chosen, batch-size-dependent hyperparameters, Adam converges, whereas under small-$β_2$ regimes it can diverge. Second, we investigate why Adam substantially outperforms SGD on Transformers through Hessian structure. We find that the Hessian evolves toward a near-block-diagonal form along training, accompanied by strong block heterogeneity. We prove that this structure makes Adam's diagonal preconditioner effective. We further show that this special Hessian structure originates from consecutive multiplications of large matrix variables, and we provide a rigorous analysis based on random matrix theory. Finally, these insights motivate Adam-mini, a new optimizer that reduces Adam's memory footprint by 50\% while preserving its performance. Our results also have broader implications beyond Adam: they reveal new local structures in matrix-based nonconvex problems, and also help understand and improve recent NN optimizers, such as Muon.

Related papers

• Adam Converges in Nonsmooth Nonconvex Optimization [3.8357180714081327]
  Adam is one of the widely implemented and influential modern bias Conversions.<n>We present the first finite-time analysis for the rate of Adam, with the iTe step and without further modifications.
  arXiv  Detail & Related papers  (2026-06-21T04:00:35Z)
• MAdam: Metric-Aware Multi-Objective Adam [12.820085175292377]
  We show two systematic gaps between the solver's intent and the coupling's execution.<n>We introduce textbfMAdam (Metric-Aware Multi-Objective Adam), a drop-in wrapper that leaves both solver and unchanged.<n>MAdam consistently improves over Adam for every solver family.
  arXiv  Detail & Related papers  (2026-06-02T17:00:15Z)
• Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers [57.049014152026864]
  In the training of neural networks, adaptive moment estimation (Adam) typically converges fast but exhibits suboptimal generalization performance.<n>To enhance its ability to find flat minima, we propose its new variant named inverse Adam (InvAdam)<n>InvAdam computes element-wise multiplication of the first-order and second-order moments, while Adam computes the element-wise division of these two moments.
  arXiv  Detail & Related papers  (2026-03-07T09:15:30Z)
• Muon Outperforms Adam in Tail-End Associative Memory Learning [118.98991042050532]
  We show that Muon consistently achieves balanced learning across classes regardless of feature embeddings.<n>Our empirical observations and theoretical analyses reveal Muon's core advantage: its update rule aligns with the outer-product structure of linear associative memories.
  arXiv  Detail & Related papers  (2025-09-30T10:04:08Z)
• AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training [22.58304858379219]
  We introduce AdamS, a simple yet effective alternative to Adam for large language model (LLM) pretraining and post-training.<n>By leveraging a novel denominator, i.e., the root of weighted sum of squares of the momentum and the current gradient, AdamS eliminates the need for second-moment estimates.<n>AdamS is efficient, matching the memory and compute footprint of SGD with momentum while delivering superior optimization performance.
  arXiv  Detail & Related papers  (2025-05-22T08:16:48Z)
• CAdam: Confidence-Based Optimization for Online Learning [41.022196390765714]
  We introduce CAdam, a confidence-based optimization strategy that assesses the consistency between the momentum and the gradient for each parameter dimension before deciding on updates.<n>CAdam significantly enhances model performance compared to Adam, leading to substantial increases in the system's gross merchandise volume (GMV)<n>In large-scale A/B testing, CAdam significantly enhances model performance compared to Adam, leading to substantial increases in the system's gross merchandise volume (GMV)
  arXiv  Detail & Related papers  (2024-11-29T12:00:27Z)
• Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity [51.96093077151991]
  We derive a new convergence guarantee of Adam, with only an $L$-smooth condition and a bounded noise variance assumption. Our proof utilizes novel techniques to handle the entanglement between momentum and adaptive learning rate.
  arXiv  Detail & Related papers  (2023-10-27T09:16:58Z)
• Provable Adaptivity of Adam under Non-uniform Smoothness [79.25087082434975]
  Adam is widely adopted in practical applications due to its fast convergence. Existing convergence analyses for Adam rely on the bounded smoothness assumption. This paper studies the convergence of randomly reshuffled Adam with diminishing learning rate.
  arXiv  Detail & Related papers  (2022-08-21T14:57:47Z)
• A Novel Convergence Analysis for Algorithms of the Adam Family [105.22760323075008]
  We present a generic proof of convergence for a family of Adam-style methods including Adam, AMSGrad, Adabound, etc. Our analysis is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non- compositional optimization problems.
  arXiv  Detail & Related papers  (2021-12-07T02:47:58Z)
• Adam$^+$: A Stochastic Method with Adaptive Variance Reduction [56.051001950733315]
  Adam is a widely used optimization method for deep learning applications. We propose a new method named Adam$+$ (pronounced as Adam-plus) Our empirical studies on various deep learning tasks, including image classification, language modeling, and automatic speech recognition, demonstrate that Adam$+$ significantly outperforms Adam.
  arXiv  Detail & Related papers  (2020-11-24T09:28:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.