Reassessing Muon for Matrix Factorization

• URL: http://arxiv.org/abs/2607.13246v1
• Date: Tue, 14 Jul 2026 20:21:38 GMT
• Title: Reassessing Muon for Matrix Factorization
• Abstract summary: Muon has emerged as a strong contender in large-scale deep learning.<n>We study Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization.<n>We find that Muon does not consistently outperform AdamW in this setting.
• Score: 9.691906492780353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.

Related papers

• Musec: MomentUm SpEctral Clipping for Stable Muon-type Training [24.72819846755016]
  Muon is prone to training instability due to its spectral growth, unbounded by loss spikes and weight variations.<n>We propose MomentUmMusec, which clips singular values of momentum matrix to approximately one, preserving the underlying spectral structure.<n> Empirically, Soft Musec consistently improves training over existing Muon variants across a wide range of learning rates and model sizes.
  arXiv  Detail & Related papers  (2026-09-10T15:00:01Z)
• Muon as a Residual Connection [4.242184336943779]
  Muon has emerged as one of the most effective preservations for training large neural networks.<n>We propose a mechanistic interpretation: Muon can be understood as an implicit residual connection during training.
  arXiv  Detail & Related papers  (2026-07-01T16:12:24Z)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory [77.27772368491698]
  Spectrals such as Muon have recently shown strong empirical performance in large-scale language model training.<n>We study this question through the linear associative memory problem.<n>We show that the storage capacity of Muon significantly exceeds that of SGD.
  arXiv  Detail & Related papers  (2026-03-27T16:13:18Z)
• To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters [16.624341041698013]
  Muon has perhaps gained the highest popularity due to its superior training speed.<n>This paper investigates the potential downsides stemming from the mechanism driving this speedup.<n>Muon struggles to uncover common underlying structure across tasks, and is more prone to fitting spurious features.
  arXiv  Detail & Related papers  (2026-02-28T17:37:15Z)
• Delving into Muon and Beyond: Deep Analysis and Extensions [8.297062899157664]
  We study Muon as the p = 0 endpoint of a family of spectral transformations of the form U boldsymbolp V'.<n>We find that RMS-normalized updates yield more stable optimization than first-moment updates.<n>Our results suggest that Muon is best understood as an effective form of spectral normalization, but not a universally superior optimization method.
  arXiv  Detail & Related papers  (2026-02-04T15:40:47Z)
• Preconditioning Benefits of Spectral Orthogonalization in Muon [50.62925024212989]
  We study the effectiveness of a simplified variant of Muon in two case studies: matrix factorization and in-context learning of linear transformers.<n>Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior.
  arXiv  Detail & Related papers  (2026-01-20T00:08:31Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Muon Outperforms Adam in Tail-End Associative Memory Learning [118.98991042050532]
  We show that Muon consistently achieves balanced learning across classes regardless of feature embeddings.<n>Our empirical observations and theoretical analyses reveal Muon's core advantage: its update rule aligns with the outer-product structure of linear associative memories.
  arXiv  Detail & Related papers  (2025-09-30T10:04:08Z)
• On the Convergence Analysis of Muon [19.29806555936508]
  We present a comprehensive convergence rate analysis of Muon and its comparison with Gradient Descent (GD)<n>Our theoretical results reveal that Muon can benefit from the low-rank and approximate blockwise diagonal structure of Hessian matrices.
  arXiv  Detail & Related papers  (2025-05-29T17:58:01Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.