Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

• URL: http://arxiv.org/abs/2609.11655v2
• Date: Fri, 11 Sep 2026 06:18:18 GMT
• Title: Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
• Abstract summary: Muon is prone to training instability due to its spectral growth, unbounded by loss spikes and weight variations.<n>We propose MomentUmMusec, which clips singular values of momentum matrix to approximately one, preserving the underlying spectral structure.<n> Empirically, Soft Musec consistently improves training over existing Muon variants across a wide range of learning rates and model sizes.
• Score: 24.72819846755016
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and AdamW optimizers. Nevertheless, Muon is prone to training instability due to its spectral flattening, manifested by loss spikes and unbounded growth of model weights. Existing approaches primarily rely on weight or attention-logit clipping, which require architecture-specific modifications and do not directly address instability across all model components. We propose MomentUm SpEctral Clipping (Musec), which replaces Muon's spectral flattening with spectral clipping: rather than setting all singular values of the momentum matrix to approximately one, Musec clips singular values that exceed a threshold while preserving the underlying spectral structure of the momentum. Our strategy provides an optimizer-level, architecture-agnostic mechanism for stabilizing Muon training. Theoretically, we establish convergence guarantees for Musec in nonconvex nonsmooth stochastic optimization. Practically, we develop Soft Musec, an efficient implementation that uses a smooth spectral saturation function approximated by coupled Newton-Schulz iterations. Empirically, Soft Musec consistently improves training stability over existing Muon variants across a wide range of learning rates and model sizes, remaining stable in settings where existing Muon variants diverge while matching their performance under well-tuned configurations.

Related papers

• Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon [12.170977868230622]
  Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam.<n>We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes.<n>We provide two variants: the complete SAMuon follows the measured profile using a low-rank randomised SVD and the simplified SAMuon-lite uses a two-level approximation via rank-one power iteration.
  arXiv  Detail & Related papers  (2026-08-26T16:41:12Z)
• Reassessing Muon for Matrix Factorization [9.691906492780353]
  Muon has emerged as a strong contender in large-scale deep learning.<n>We study Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization.<n>We find that Muon does not consistently outperform AdamW in this setting.
  arXiv  Detail & Related papers  (2026-07-14T20:21:38Z)
• DMuon: Efficient Distributed Muon Training with Near-Adam Overhead [13.192517400076746]
  We present DMuon, an open-source distributed Muon implementation that integrates into existing training pipelines as a drop-in module.<n>DMuon achieves a 1.48x-3.01x speedup in end-to-end step time and a 6.85x-163.00x speedup in-step time, bringing per-step latency to near-AdamW levels.
  arXiv  Detail & Related papers  (2026-06-25T15:23:03Z)
• Muon is Not That Special: Random or Inverted Spectra Work Just as Well [50.969177887027115]
  We demonstrate that precise geometric structure is not key factor affecting performance.<n>We introduce Freon, a family of geometrics based on Schatten (quasi-size)norms.<n>We also introduce Kaon, an absurd that replaces singular values with random noise.
  arXiv  Detail & Related papers  (2026-05-11T19:42:48Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning [72.62839712454196]
  Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improveimation.<n>We argue that this "egalitarian" constraint is suboptimal for Deep Neural Networks, where the curvature spectrum is known to be highly heavy-tailed and ill-conditioned.<n>We propose textbfMousse (textbfMuon textbfOptimization textbfUtilizing textbfShampoo's textbfStruct
  arXiv  Detail & Related papers  (2026-03-10T14:03:49Z)
• NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training [50.27276603708547]
  We show that despite imposing full-rank updates, Muon-trained models exhibit pronounced low-rank structure in their weight matrices and are readily compressible under standard pipelines.<n>We propose NuMuon, which augments Muon with a nuclear-norm constraint on the update direction, further constraining the learned weights toward low-rank structure.
  arXiv  Detail & Related papers  (2026-03-04T00:10:14Z)
• Controlled LLM Training on Spectral Sphere [76.60985966206746]
  We introduce the textbfSpectral Sphere algorithm (SSO), which enforces strict module-wise spectral constraints on both weights and their updates.<n>We observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations.
  arXiv  Detail & Related papers  (2026-01-13T09:59:47Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• AdaMuon: Adaptive Muon Optimizer [11.281916426508216]
  AdaMuon combines element-wise adaptivity with orthogonal updates for large-scale neural network training.<n>AdaMuon maintains stability but can surpass Adam by more than 40% training efficiency in large-scale scenarios.
  arXiv  Detail & Related papers  (2025-07-15T05:49:37Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.