Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

• URL: http://arxiv.org/abs/2606.23676v1
• Date: Mon, 22 Jun 2026 17:58:52 GMT
• Title: Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
• Abstract summary: AdamW is the de facto theory for training large language models.<n>Recent work shows that sign-based gradients such as Lion and Muon achieve sharp heavy-tailed rates.<n>Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction?
• Score: 43.39716211464324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction? We formulate this as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism showing how denominator memory can hide large gradients.

Related papers

• Can Muon Fine-tune Adam-Pretrained Models? [0.5735035463793009]
  Most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to a mismatch.<n>We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength.<n>We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning.
  arXiv  Detail & Related papers  (2026-05-11T12:34:20Z)
• Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise [43.39716211464324]
  Sign-based optimization algorithms such as Lion and Muon have recently demonstrated superior empirical performance over AdamW.<n>In this paper, we aim to bridge the gap between theory and practice through the lens of heavy-tailed gradient noise.
  arXiv  Detail & Related papers  (2026-02-07T07:47:14Z)
• Conda: Column-Normalized Adam for Training Large Language Models Faster [70.66067959375748]
  Column-Normalized Adam (Conda) is a novel approach to large language models (LLMs)<n>Conda projects updates into a subspace and applies column-wise second moment normalization based on the projected gradients.<n>Experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training.
  arXiv  Detail & Related papers  (2025-09-29T02:58:19Z)
• Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed [83.8485684139678]
  Methods with adaptive stepsizes, such as AdaGrad and Adam, are essential for training modern Deep Learning models.<n>We show that AdaGrad/Adam methods can have bad high-probability convergence if the noise is heavy-tailed.
  arXiv  Detail & Related papers  (2024-06-06T18:49:10Z)
• On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions [4.189643331553922]
  Adaptive Momentum Estimation (Adam) algorithm is highly effective in various deep learning tasks.<n>We show that Adam can find a stationary point variance with a rate in high iterations under this general noise model.
  arXiv  Detail & Related papers  (2024-02-06T13:19:26Z)
• Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be [16.170888329408353]
  We show that the behavior of Adam with large batches is similar to sign descent with momentum. We present evidence thatity and heavy-tailed noise are not major factors in the performance gap between SGD and Adam.
  arXiv  Detail & Related papers  (2023-04-27T05:41:13Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)
• Identifying Hard Noise in Long-Tailed Sample Distribution [71.8462682319137]
  We introduce Noisy Long-Tailed Classification (NLT)<n>Most de-noising methods fail to identify the hard noises.<n>We design an iterative noisy learning framework called Hard-to-Easy (H2E)
  arXiv  Detail & Related papers  (2022-07-27T09:03:03Z)
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance [76.54300276636982]
  Noise in gradient descent provides a crucial implicit regularization effect for training over parameterized models. We show that parameter-dependent noise -- induced by mini-batches or label perturbation -- is far more effective than Gaussian noise. Our analysis reveals that parameter-dependent noise introduces a bias towards local minima with smaller noise variance, whereas spherical Gaussian noise does not.
  arXiv  Detail & Related papers  (2020-06-15T18:31:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.