Muon Meets Mamba: Spectral Optimization for State Space Models

• URL: http://arxiv.org/abs/2608.03941v1
• Date: Tue, 04 Aug 2026 17:10:47 GMT
• Title: Muon Meets Mamba: Spectral Optimization for State Space Models
• Abstract summary: We compare Muon with AdamW on Mamba-2 130M under a controlled protocol that varies only which weight groups are trained with Muon.<n>Muon on the output projection alone beats Muon on the input projection or on both.<n> Conditioning does not explain the gain. Muon lowers the condition number of whichever projection it trains, but the better-conditioned input projection is not the one that helps.
• Score: 39.146761527401424
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Muon is a recent optimizer that orthogonalizes the update to each weight matrix with a Newton-Schulz iteration, which performs steepest descent under the spectral norm. Almost all the evidence for it comes from Transformer models, and its behavior on state-space models is largely unreported. We compare Muon with AdamW on Mamba-2 130M under a controlled protocol that varies only which weight groups are trained with Muon. The benefit is localized. Muon on the output projection alone beats Muon on the input projection or on both. The advantage is mainly one of token efficiency. It holds on two corpora and two token budgets, and persists when training continues well past the compute-optimal point. Conditioning does not explain the gain. Muon lowers the condition number of whichever projection it trains, but the better-conditioned input projection is not the one that helps.

Related papers

• Why Muon Outperforms Adam: A Curvature Perspective [49.85900116602357]
  Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
  arXiv  Detail & Related papers  (2026-06-03T09:40:30Z)
• LionMuon: Alternating Spectral and Sign Descent for Efficient Training [48.69598464267559]
  We propose LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged cost.<n>It alternates between Lion's and Muon's updates on a fixed period P, sharing a single dual-EMA momentum buffer between them.<n>A single-EMA variant, SignMuon, by itself already outperforms pure Muon.
  arXiv  Detail & Related papers  (2026-05-19T13:07:59Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training [50.27276603708547]
  We show that despite imposing full-rank updates, Muon-trained models exhibit pronounced low-rank structure in their weight matrices and are readily compressible under standard pipelines.<n>We propose NuMuon, which augments Muon with a nuclear-norm constraint on the update direction, further constraining the learned weights toward low-rank structure.
  arXiv  Detail & Related papers  (2026-03-04T00:10:14Z)
• Muon+: Towards Better Muon via One Additional Normalization Step [18.816463168231618]
  We propose a simple yet effective enhancement to Muon, namely Muon+.<n>We demonstrate the effectiveness of Muon+ through extensive pre-training experiments across a wide range of model scales and architectures.
  arXiv  Detail & Related papers  (2026-02-25T04:04:00Z)
• MuonAll: Muon Variant for Efficient Finetuning of Large Language Models [0.0]
  We introduce MuonAll, which incorporates all the parameters inside Muon by transforming into 2D matrices.<n>We conduct extensive finetuning experiments across publicly available language models with model sizes upto half billion parameters.
  arXiv  Detail & Related papers  (2025-11-08T17:45:20Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.