Why Muon Outperforms Adam: A Curvature Perspective

• URL: http://arxiv.org/abs/2606.04662v1
• Date: Wed, 03 Jun 2026 09:40:30 GMT
• Title: Why Muon Outperforms Adam: A Curvature Perspective
• Abstract summary: Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
• Score: 49.85900116602357
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon improves training efficiency over Adam in large language-model training by about two times, but the local geometric source of this advantage remains unclear. Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective. First, we apply a second-order Taylor approximation to the training landscape and show that Muon achieves a larger one-step loss decrease than Adam at matched validation loss. The two optimizers have comparable first-order gains, but Muon consistently incurs a smaller second-order curvature penalty. Second, we decompose this curvature penalty into the squared update norm and Normalized Directional Sharpness (NDS). We find that Muon and Adam have comparable update norms, so Muon's smaller curvature penalty is driven by lower NDS, not update scale. Third, we study how training data and model structure shape Muon's NDS advantage. Using Zipf-Probabilistic Context-Free Grammar (PCFG) data with controlled imbalance, we show that data imbalance amplifies Muon's NDS advantage over Adam. A within-/cross-layer decomposition further shows that, in the middle and late stages of training, Muon's lower NDS is mainly sustained by smaller within-layer curvature. Beyond empirical evidence, we analyze stylized quadratic problems with heterogeneous curvature and gradient alignment toward high-curvature modes. We prove that Muon attains a smaller average NDS than GD by balancing update energy across curvature groups; when curvature heterogeneity is sufficiently strong, this also yields lower local quadratic loss after the same number of steps.

Related papers

• Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon [12.170977868230622]
  Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam.<n>We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes.<n>We provide two variants: the complete SAMuon follows the measured profile using a low-rank randomised SVD and the simplified SAMuon-lite uses a two-level approximation via rank-one power iteration.
  arXiv  Detail & Related papers  (2026-08-26T16:41:12Z)
• DynMuon: A Dynamic Spectral Shaping View of Muon [12.175075916077702]
  We consider a class of Muon-like updates, where we replace the update $M$ with $Up Vtop$ for some parameter $p$.<n>We propose DynMuon, an efficient dynamic spectral shaping method that schedules $p$ from positive to mildly negative over training.
  arXiv  Detail & Related papers  (2026-05-16T18:30:11Z)
• Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence [15.754779716816174]
  We show that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest.<n>We recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor.
  arXiv  Detail & Related papers  (2026-05-13T06:54:01Z)
• Muown: Row-Norm Control for Muon Optimization [41.541871767555484]
  Muon is a strong competitor to AdamW for language model pre-training.<n>Muown improves perplexity over Muon, SOAP, AdamW, and Lion.
  arXiv  Detail & Related papers  (2026-05-11T16:26:06Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Muon Outperforms Adam in Tail-End Associative Memory Learning [118.98991042050532]
  We show that Muon consistently achieves balanced learning across classes regardless of feature embeddings.<n>Our empirical observations and theoretical analyses reveal Muon's core advantage: its update rule aligns with the outer-product structure of linear associative memories.
  arXiv  Detail & Related papers  (2025-09-30T10:04:08Z)
• Conda: Column-Normalized Adam for Training Large Language Models Faster [70.66067959375748]
  Column-Normalized Adam (Conda) is a novel approach to large language models (LLMs)<n>Conda projects updates into a subspace and applies column-wise second moment normalization based on the projected gradients.<n>Experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training.
  arXiv  Detail & Related papers  (2025-09-29T02:58:19Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.