MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models

• URL: http://arxiv.org/abs/2605.19619v1
• Date: Tue, 19 May 2026 09:56:27 GMT
• Title: MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models
• Abstract summary: We study generalization error of the Muon based on algorithmic stability and mathematical induction.<n>We propose an effective mixed Muon (MiMuon) by cautiously using gradient, which is a hybrid of Muon and momentum-based SGDs.<n>Our MiMuon algorithm has the same convergence rate of $O(frac1NTbig)$ as the Muon algorithm.
• Score: 45.11415579822849
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Matrix-structured parameters frequently appear in many artificial intelligence models such as large language models. More recently, an efficient Muon optimizer is designed for matrix parameters of large-scale models, and shows markedly faster convergence than the vector-wise algorithms. Although some works have begun to study convergence properties (i.e., optimization error) of the Muon optimizer, its generalization properties (i.e., generalization error) is still not established. Thus, in this paper, we study generalization error of the Muon optimizer based on algorithmic stability and mathematical induction, and prove that the Muon has a generalization error of $O\big(\frac{1}{Nκ^{T}}\big)$, where $N$ is training sample size, and $T$ denotes iteration number, and $κ>0$ denotes minimum difference between singular values of gradient estimate. To enhance generalization of the Muon, we propose an effective mixed Muon (MiMuon) optimizer by cautiously using orthogonalization of gradient, which is a hybrid of Muon and momentum-based SGD optimizers. Then we prove that our MiMuon optimizer has a lower generalization error of $O\big(\frac{1}{N}\big)$ than $O\big(\frac{1}{Nκ^{T}}\big)$ of Muon optimizer, since $κ$ generally is very small. Meanwhile, we also studied the convergence properties of our MiMuon algorithm, and prove that our MiMuon algorithm has the same convergence rate of $O(\frac{1}{T^{1/4}})$ as the Muon algorithm. Some numerical experimental results on training large models including Qwen3-0.6B and YOLO26m demonstrate efficiency of the MiMuon optimizer.

Related papers

• Approximate Muon with low-rank adapters [7.692230106198559]
  Muon shows clear benefits versus alternatives when pretraining neural networks.<n>However, it is used less frequently for parameter-efficient fine-tuning.<n>We show that using Muon for low-rank fine-tuning provides moderate performance improvements.
  arXiv  Detail & Related papers  (2026-08-14T17:07:37Z)
• Muon$^p$: Muon with Fractional Spectral Powers [3.223825674658464]
  Muon is an increasingly widely used gradient that replaces a $G=USVtop$ with its polar factor $UVtop$.<n>We introduce Muon$p$ that instead uses fractional spectral-power updates $USpVtop$ for rational $pin(0,1)$.<n>Our results reveal important insights on when preserving the singular spectrum can bring significant gains, and introduce a principled way to achieve them.
  arXiv  Detail & Related papers  (2026-06-11T19:48:12Z)
• Muon is Not That Special: Random or Inverted Spectra Work Just as Well [50.969177887027115]
  We demonstrate that precise geometric structure is not key factor affecting performance.<n>We introduce Freon, a family of geometrics based on Schatten (quasi-size)norms.<n>We also introduce Kaon, an absurd that replaces singular values with random noise.
  arXiv  Detail & Related papers  (2026-05-11T19:42:48Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• LiMuon: Light and Fast Muon Optimizer for Large Models [45.11415579822849]
  We propose a useful Muon for training large models.<n>Our LiMuon has a lower memory than the current Muon and its variants.<n>We prove that our LiMuon has a sample $O(epsilon-3)$ under the generalized smooth condition.
  arXiv  Detail & Related papers  (2025-09-18T02:49:27Z)
• Muon Optimizes Under Spectral Norm Constraints [12.29696026957078]
  We show that Muon implicitly solves an optimization problem that enforces a constraint on the spectral norm of weight matrices.<n>This perspective allows for the exploration of a broader class of implicitly regularized and constrained optimization algorithms.
  arXiv  Detail & Related papers  (2025-06-18T01:32:39Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.