Beyond a Single Explanation of the Adam--SGD Gap

• URL: http://arxiv.org/abs/2606.14259v1
• Date: Fri, 12 Jun 2026 08:43:53 GMT
• Title: Beyond a Single Explanation of the Adam--SGD Gap
• Abstract summary: Previous work has identified several factors that can contribute to the performance gap between Adam and SGD.<n>Our results suggest that no single factor consistently explains the Adam--SGD gap.
• Score: 27.821705738798986
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties. Yet these explanations are often studied in isolation, leaving their relative importance unclear. In this work, we revisit these hypotheses through a controlled empirical study across vision, language, genomics, and graph tasks, spanning modern and classical architectures, and carefully designed training setups. Our results suggest that no single factor consistently explains the Adam--SGD gap. For instance, the Adam advantage can (1) persist under a uniform vocabulary distribution yet nearly disappear under a heavy-tailed one; (2) reverse in favor of SGD in softmax-attention models; and (3) become larger under soft architectural modifications, e.g., when ReLU is replaced by a GeLU nonlinearity. This suggests that the gap arises from nontrivial data and architecture interactions, rather than from a single common factor. Yet, we observe a pattern across our settings: a \emph{crossover batch size} at which the relative advantage shifts from SGD to Adam as the batch size scales. These empirical results are captured by our theoretical gap model, which predicts this batch-size-dependent crossover. Our perspective helps reconcile several existing hypotheses while offering practical insights across domains.

Related papers

• Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling [36.106114687828395]
  Adam is known to perform significantly better than Gradient Descent (SGD) in language models.<n>We exhaustively study how momentum, gradient clipping, and batch size affect the gap between SGD and Adam.
  arXiv  Detail & Related papers  (2025-06-14T15:37:31Z)
• The Rich and the Simple: On the Implicit Bias of Adam and SGD [26.722625797251553]
  Adam is the de facto optimization algorithm for several deep learning applications.<n>In practice, neural networks (NNs) trained with (stochastic) gradient descent (GD) are known to exhibit simplicity bias.<n>We show that Adam is more resistant to such simplicity bias.
  arXiv  Detail & Related papers  (2025-05-29T21:46:12Z)
• A Simple and Generalist Approach for Panoptic Segmentation [57.94892855772925]
  We propose a simple generalist framework based on a deep encoder - shallow decoder architecture with per-pixel prediction.<n>We show that this is due to imbalance during training and propose a novel method for reducing it.<n>Our method achieves panoptic quality (PQ) of 55.1 on the challenging MS-COCO dataset.
  arXiv  Detail & Related papers  (2024-08-29T13:02:12Z)
• Linking Robustness and Generalization: A k* Distribution Analysis of Concept Clustering in Latent Space for Vision Models [56.89974470863207]
  This article uses the k* Distribution, a local neighborhood analysis method, to examine the learned latent space at the level of individual concepts. We introduce skewness-based true and approximate metrics for interpreting individual concepts to assess the overall quality of vision models' latent space.
  arXiv  Detail & Related papers  (2024-08-17T01:43:51Z)
• Predictive Churn with the Set of Good Models [61.00058053669447]
  This paper explores connections between two seemingly unrelated concepts of predictive inconsistency.<n>The first, known as predictive multiplicity, occurs when models that perform similarly produce conflicting predictions for individual samples.<n>The second concept, predictive churn, examines the differences in individual predictions before and after model updates.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be [16.170888329408353]
  We show that the behavior of Adam with large batches is similar to sign descent with momentum. We present evidence thatity and heavy-tailed noise are not major factors in the performance gap between SGD and Adam.
  arXiv  Detail & Related papers  (2023-04-27T05:41:13Z)
• From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent [50.4531316289086]
  Gradient Descent (SGD) has been the method of choice for learning large-scale non-root models. An overarching paper is providing general conditions SGD converges, assuming that GF on the population loss converges. We provide a unified analysis for GD/SGD not only for classical settings like convex losses, but also for more complex problems including Retrieval Matrix sq-root.
  arXiv  Detail & Related papers  (2022-10-13T03:55:04Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Variational Causal Networks: Approximate Bayesian Inference over Causal Structures [132.74509389517203]
  We introduce a parametric variational family modelled by an autoregressive distribution over the space of discrete DAGs. In experiments, we demonstrate that the proposed variational posterior is able to provide a good approximation of the true posterior.
  arXiv  Detail & Related papers  (2021-06-14T17:52:49Z)
• Fundamental Limits and Tradeoffs in Invariant Representation Learning [99.2368462915979]
  Many machine learning applications involve learning representations that achieve two competing goals. Minimax game-theoretic formulation represents a fundamental tradeoff between accuracy and invariance. We provide an information-theoretic analysis of this general and important problem under both classification and regression settings.
  arXiv  Detail & Related papers  (2020-12-19T15:24:04Z)
• AdaSGD: Bridging the gap between SGD and Adam [14.886598905466604]
  We identify potential differences in performance between SGD and Adam. We demonstrate how AdaSGD combines the benefits both SGD Adam and SGD non- descent.
  arXiv  Detail & Related papers  (2020-06-30T05:44:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.