MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training

• URL: http://arxiv.org/abs/2605.26842v1
• Date: Tue, 26 May 2026 10:56:20 GMT
• Title: MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
• Abstract summary: We present MONA, an acceleration term that bridges Muon's orthogonalization framework with curvature-aware acceleration.<n>We provide a detailed convergence analysis for MONA, showing that the acceleration term enables escape from sharp minima.<n>We evaluate it on general capability, mathematical reasoning, and code generation benchmarks, where MONA achieves SOTA performance.
• Score: 32.15839901598669
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Muon can become trapped in sharp local minima. In this work, we present MONA, an optimizer that bridges Muon's orthogonalization framework with curvature-aware acceleration. MONA adds an acceleration term directly into Muon's gradient processing pipeline. This term is calculated from the exponential moving average of gradient differences. We provide a detailed convergence analysis for MONA, showing that the acceleration term enables escape from sharp minima while preserving Muon's spectral-norm regularization. Empirically, MONA achieves better convergence and downstream task performance compared to both Muon and AdamW across three scales of Mixture-of-Experts pretraining, spanning from 1B to 68B parameters, with the largest model trained on 1 trillion tokens. Furthermore, we conduct supervised fine-tuning on the MOE-68B-A3B model and evaluate it on general capability, mathematical reasoning, and code generation benchmarks, where MONA achieves SOTA performance.

Related papers

• Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon [12.170977868230622]
  Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam.<n>We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes.<n>We provide two variants: the complete SAMuon follows the measured profile using a low-rank randomised SVD and the simplified SAMuon-lite uses a two-level approximation via rank-one power iteration.
  arXiv  Detail & Related papers  (2026-08-26T16:41:12Z)
• DMuon: Efficient Distributed Muon Training with Near-Adam Overhead [13.192517400076746]
  We present DMuon, an open-source distributed Muon implementation that integrates into existing training pipelines as a drop-in module.<n>DMuon achieves a 1.48x-3.01x speedup in end-to-end step time and a 6.85x-163.00x speedup in-step time, bringing per-step latency to near-AdamW levels.
  arXiv  Detail & Related papers  (2026-06-25T15:23:03Z)
• Understanding and Accelerating the Training of Masked Diffusion Language Models [104.60946175023008]
  Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling.<n>We show how to accelerate standard MDM training while maintaining its final performance.<n>We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T05:29:38Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory [77.27772368491698]
  Spectrals such as Muon have recently shown strong empirical performance in large-scale language model training.<n>We study this question through the linear associative memory problem.<n>We show that the storage capacity of Muon significantly exceeds that of SGD.
  arXiv  Detail & Related papers  (2026-03-27T16:13:18Z)
• Variance-Adaptive Muon: Accelerating LLM Pretraining with NSR-Modulated and Variance-Scaled Momentum [19.385264518362472]
  Large Language Models (LLMs) achieve competitive performance across diverse natural language processing (NLP) tasks.<n>We propose two variants of Muon, Muon-NSR and Muon-VS, which apply variance-adaptive normalization to momentum.<n>Experiments on GPT-2 and LLaMA pretraining demonstrate that our proposed methods accelerate convergence and consistently achieve lower validation loss than both competitive, well-tuned AdamW and Muon baselines.
  arXiv  Detail & Related papers  (2026-01-21T02:41:56Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• MARS-M: When Variance Reduction Meets Matrices [47.405031764674014]
  Matrix-based preconditioneds have been shown to be more efficient than scalar-based preconditioneds for large-scale neural networks.<n>We introduce MARS-M, a new technique that integrates the variance reduction technique in MARS with Muon.<n>Our empirical results on language modeling and computer vision tasks demonstrate that MARS-M consistently yields lower losses and improved performance across various downstream benchmarks.
  arXiv  Detail & Related papers  (2025-10-20T16:49:22Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Muon: Training and Trade-offs with Latent Attention and MoE [4.500362688166346]
  We present a comprehensive theoretical and empirical study of the Muon for training transformers only with a small to medium decoder (30M - 200M parameters)<n>We provide rigorous theoretical analysis including: (i)showing the convergence rate under standard assumptions, (ii) spectral regularization properties that prevent gradient explosion, (iii) connection to natural gradient descent on the Stiefel manifold, and (iv) equivalence to steepest gradient descent under the spectral norm.
  arXiv  Detail & Related papers  (2025-09-29T07:51:06Z)
• AdaMuon: Adaptive Muon Optimizer [11.281916426508216]
  AdaMuon combines element-wise adaptivity with orthogonal updates for large-scale neural network training.<n>AdaMuon maintains stability but can surpass Adam by more than 40% training efficiency in large-scale scenarios.
  arXiv  Detail & Related papers  (2025-07-15T05:49:37Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.