How Much Orthogonalization Does Muon Need?

• URL: http://arxiv.org/abs/2606.00371v1
• Date: Fri, 29 May 2026 21:26:25 GMT
• Title: How Much Orthogonalization Does Muon Need?
• Abstract summary: Muons improve neural-network training by replacing ill-conditioned momentum updates with approximately semi-orthogonal updates.<n>We use a relaxed cubic Newton--Schulz schedule derived directly for Muon's low precision singular value band.<n>We find that training quality is not governed monotonically by polar-decomposition accuracy.
• Score: 20.05604748384174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon optimizers improve neural-network training by replacing ill-conditioned momentum updates with approximately semi-orthogonal updates. This motivates a practical question: how much orthogonalization does Muon actually require? We study this question using a relaxed cubic Newton--Schulz schedule derived directly for Muon's low precision singular value band. The resulting five-step cubic construction uses ten dominant matrix multiplications, compared with fifteen for five quintic Newton--Schulz iterations. The cubic schedule is not intended as a more accurate polar solver; instead, it is a principled low-cost variant that lets us probe the relation between polar accuracy, spectral shaping, and training quality. Across synthetic diagnostics, NanoGPT ablations, and training experiments on hybrid MoE/Mamba models, we find that training quality is not governed monotonically by polar-decomposition accuracy: truncated Polar Express, Muon-Jordan, cubic Newton--Schulz, and an explicit FP32 SVD polar factor can reach nearly indistinguishable final loss on GPT-2 Small, and cubic5 matches the Muon-Jordan quintic update within about $10^{-3}$ validation loss on hybrid MoE/Mamba models with one billion to four billion parameters. These results support cubic5 as a practical low-cost Muon orthogonalization variant, with empirical evidence of training-quality parity in the settings tested.

Related papers

• Why Muon Outperforms Adam: A Curvature Perspective [49.85900116602357]
  Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
  arXiv  Detail & Related papers  (2026-06-03T09:40:30Z)
• DynMuon: A Dynamic Spectral Shaping View of Muon [12.175075916077702]
  We consider a class of Muon-like updates, where we replace the update $M$ with $Up Vtop$ for some parameter $p$.<n>We propose DynMuon, an efficient dynamic spectral shaping method that schedules $p$ from positive to mildly negative over training.
  arXiv  Detail & Related papers  (2026-05-16T18:30:11Z)
• MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization [7.243820893114047]
  We propose a low-bit Muon training framework built on the principle of directional fidelity optimization.<n>MuonQ at 4-bit precision closely matches full-precision Muon in both training loss and downstream task accuracy.<n>Our code is available at https://github.com/YupSueng/MuonQ.
  arXiv  Detail & Related papers  (2026-05-12T01:31:32Z)
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning [262.0414794728708]
  We describe the pre-training, post-training, and quantization of Nemotron 3 Super.<n>Nemotron 3 Super is a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model.
  arXiv  Detail & Related papers  (2026-04-14T07:02:32Z)
• Communication-Efficient Gluon in Federated Learning [89.94131981227609]
  We investigate Gluon, which is an extension of Muon under the more general layer-wise $(L0, L1)$-smooth setting.<n>In order to reduce the compression error, we employ the variance reduced technique in SARAH in our compressed methods.
  arXiv  Detail & Related papers  (2026-04-12T15:30:28Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• The Newton-Muon Optimizer [2.1583596008762935]
  We introduce a surrogate model that sheds new light on the design iteration of Muon.<n>New optimization method, which we refer to as Newton-Muon, shows that standard Muon can be interpreted as an implicit Newton-type method.
  arXiv  Detail & Related papers  (2026-04-01T23:29:08Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.