LionMuon: Alternating Spectral and Sign Descent for Efficient Training

• URL: http://arxiv.org/abs/2605.19811v1
• Date: Tue, 19 May 2026 13:07:59 GMT
• Title: LionMuon: Alternating Spectral and Sign Descent for Efficient Training
• Abstract summary: We propose LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged cost.<n>It alternates between Lion's and Muon's updates on a fixed period P, sharing a single dual-EMA momentum buffer between them.<n>A single-EMA variant, SignMuon, by itself already outperforms pure Muon.
• Score: 48.69598464267559
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In large-scale optimization, the cheapness and effectiveness of update steps are the most crucial factors for a successful optimizer. Sign-based optimizers like Lion or Signum produce cheap per-step updates, whereas Muon's spectral matrix-sign update gives a much stronger direction at a substantially higher per-step cost. In this work, we propose LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged iteration cost, similar to sign-based methods. It alternates between Lion's and Muon's updates on a fixed period P, sharing a single dual-EMA momentum buffer between them. The optimizer state memory therefore matches Lion and is exactly half of AdamW's. A simpler single-EMA variant, SignMuon, by itself already outperforms pure Muon. At P = 2, LionMuon Pareto-dominates Muon, Lion, Signum, and AdamW on every dataset and architecture we tested at 124M model size, reaching lower validation loss at lower compute, and the same advantage persists at 355M and 720M scale. On the theory side, we prove sharp complexity bounds under heavy-tailed noise which are governed by period-averaged smoothness and noise that interpolate between Muon's and Lion's constants. These bounds predict the compute-optimal period and the conditions under which LionMuon outruns Muon and Lion. Code: https://github.com/brain-lab-research/lion-muon

Related papers

• Muon Meets Mamba: Spectral Optimization for State Space Models [39.146761527401424]
  We compare Muon with AdamW on Mamba-2 130M under a controlled protocol that varies only which weight groups are trained with Muon.<n>Muon on the output projection alone beats Muon on the input projection or on both.<n> Conditioning does not explain the gain. Muon lowers the condition number of whichever projection it trains, but the better-conditioned input projection is not the one that helps.
  arXiv  Detail & Related papers  (2026-08-04T17:10:47Z)
• Why Muon Outperforms Adam: A Curvature Perspective [49.85900116602357]
  Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
  arXiv  Detail & Related papers  (2026-06-03T09:40:30Z)
• MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models [45.11415579822849]
  We study generalization error of the Muon based on algorithmic stability and mathematical induction.<n>We propose an effective mixed Muon (MiMuon) by cautiously using gradient, which is a hybrid of Muon and momentum-based SGDs.<n>Our MiMuon algorithm has the same convergence rate of $O(frac1NTbig)$ as the Muon algorithm.
  arXiv  Detail & Related papers  (2026-05-19T09:56:27Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Muon+: Towards Better Muon via One Additional Normalization Step [18.816463168231618]
  We propose a simple yet effective enhancement to Muon, namely Muon+.<n>We demonstrate the effectiveness of Muon+ through extensive pre-training experiments across a wide range of model scales and architectures.
  arXiv  Detail & Related papers  (2026-02-25T04:04:00Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Muon Outperforms Adam in Tail-End Associative Memory Learning [118.98991042050532]
  We show that Muon consistently achieves balanced learning across classes regardless of feature embeddings.<n>Our empirical observations and theoretical analyses reveal Muon's core advantage: its update rule aligns with the outer-product structure of linear associative memories.
  arXiv  Detail & Related papers  (2025-09-30T10:04:08Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)
• Symbolic Discovery of Optimization Algorithms [132.62397077095787]
  We use efficient search techniques to explore an infinite and sparse program space. Our method discovers a simple and effective optimization algorithm, $textbfLion$. Lion is successfully deployed in production systems such as Google search ads CTR model.
  arXiv  Detail & Related papers  (2023-02-13T20:27:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.