Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates

• URL: http://arxiv.org/abs/2605.17787v1
• Date: Mon, 18 May 2026 03:09:50 GMT
• Title: Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
• Abstract summary: gradient descent (SGD) performs significantly worse than adaptives such as Adam in pre-training Large Language Models.<n>In this work, we attribute a large part of the discrepancy to SGD's inability to sustain learning rates comparable to Adam's much larger effective learning rates.<n>We show that simple clipping mechanisms that stabilize SGD at large learning rates enable it to recover most of Adam's performance.
• Score: 14.645982276552592
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptive optimizers such as Adam in pre-training Large Language Models (LLMs). Yet the underlying reason for this gap remains unclear. In this work, we attribute a large part of the discrepancy to SGD's inability to sustain learning rates comparable to Adam's much larger effective learning rates. Through empirical and theoretical analysis of LLM pre-training dynamics, we identify that training is characterized by small gradient norms and large weight-to-gradient ratios, an effect that becomes more pronounced with larger batch sizes typical in pre-training, necessitating such large effective learning rates. However, we find that output-layer gradient magnitudes become highly uneven across token classes, and that large gradient spikes frequently occur during training. Together, these effects severely restrict the admissible learning rate of SGD. Guided by this understanding, we show that simple clipping mechanisms that stabilize SGD at large learning rates enable it to recover most of Adam's performance. In our large-scale experiments, the validation loss gap between large-learning-rate SGD and Adam shrinks from more than 50% to only about 3.5% when pre-training a 1B-parameter LLaMA model with a 1M-token batch size.

Related papers

• On the Nonlinearity of Learning Rate Scaling for LLM Training [18.557199144090017]
  Learning-rate transfer can reduce the cost of training large language models.<n>Existing approaches often assume that the optimal learning rate follows a log-linear scaling law in data scale and model size.
  arXiv  Detail & Related papers  (2026-06-28T02:42:47Z)
• Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs [21.242959630751663]
  We show that reinforcement learning can be substantially more parameter-efficient than previously recognized.<n>Experiments demonstrate that the substantially more memory-efficient SGD matches or even outperforms AdamW in RL for LLMs.
  arXiv  Detail & Related papers  (2026-02-07T23:25:26Z)
• Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMs [60.68927774057402]
  We show, for the first time, that a lower simplicity bias induces a better generalization.<n>Motivated by this insight, we demonstrate that the training data distribution by upsampling or augmenting examples learned later in training similarly reduces SB and leads to improved generalization.<n>Our strategy improves the performance of multiple language models including Phi2-2.7B, Llama3.2-1B, Gemma3-1B-PT, Qwen3-0.6B-Base-achieving relative accuracy gains up to 18% when fine-tuned with AdamW and Muon.
  arXiv  Detail & Related papers  (2026-01-31T07:40:36Z)
• Learn More, Forget Less: A Gradient-Aware Data Selection Approach for LLM [51.21051698747157]
  We propose a self-adaptive gradient-aware data selection approach (GrADS) for supervised fine-tuning of large language models (LLMs)<n>Specifically, we design self-guided criteria that leverage the magnitude and statistical distribution of gradients to prioritize examples that contribute the most to the model's learning process.<n>Through extensive experimentation with various LLMs across diverse domains such as medicine, law, and finance, GrADS has demonstrated significant efficiency and cost-effectiveness.
  arXiv  Detail & Related papers  (2025-11-07T08:34:50Z)
• Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling [75.36692892951018]
  Increasing the batch size during training is a promising strategy to accelerate large language model pretraining.<n>This work develops a principled framework for batch-size scheduling.<n>It introduces Seesaw: whenever a standard scheduler would halve the learning rate, Seesaw instead multiplies it by $1/sqrt2$ and doubles the batch size.
  arXiv  Detail & Related papers  (2025-10-16T14:17:38Z)
• FOCUS: First Order Concentrated Updating Scheme [9.278468089636547]
  Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further.<n>We show that when gradient query noise is high relative to the valley's sharpness, Adam's performance falls behind that of Signum because Adam reduces the effective step size too drastically.<n>We develop FOCUS, which enhances Signum by incorporating attraction toward moving averaged parameters, allowing it to handle noise better while maintaining larger step sizes.
  arXiv  Detail & Related papers  (2025-01-21T16:03:42Z)
• SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training [60.9776082805359]
  Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks, yet their training remains highly resource-intensive and susceptible to training instability.<n>This paper presents a comprehensive investigation into gradient spikes observed during LLM training, revealing their prevalence across multiple architectures and datasets.<n>We propose Spike-Aware Adam with Momentum Reset, a novel designed to counteract gradient spikes through momentum reset and spike-aware clipping.
  arXiv  Detail & Related papers  (2025-01-12T15:21:22Z)
• No More Adam: Learning Rate Scaling at Initialization is All You Need [13.892699813809857]
  SGD-SaI is a simple yet effective enhancement to gradient descent with momentum (SGDM)<n>By adjusting learning rates without relying on adaptive second-order momentum, SGD-SaI helps prevent training imbalances from the very first.<n>Despite its simplicity and efficiency, SGD-SaI consistently matches or outperforms AdamW in training a variety of Transformer-based tasks.
  arXiv  Detail & Related papers  (2024-12-16T13:41:37Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)
• On Avoiding Local Minima Using Gradient Descent With Large Learning Rates [48.42061343682201]
  We argue that a complete understanding of the mechanics leading to descent (GD)'s success may indeed require considering effects of using a large step size. We show that GD with large step size follows a different trajectory than GD with a small step size, leading to convergence to the global minimum. We also demonstrate the difference in trajectories for small and large learning rates when GD is applied on a neural network.
  arXiv  Detail & Related papers  (2022-05-30T14:35:32Z)
• On the Generalization Benefit of Noise in Stochastic Gradient Descent [34.127525925676416]
  It has long been argued that minibatch gradient descent can generalize better than large batch gradient descent in deep neural networks. We show that small or moderately large batch sizes can substantially outperform very large batches on the test set.
  arXiv  Detail & Related papers  (2020-06-26T16:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.