Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

• URL: http://arxiv.org/abs/2605.13079v1
• Date: Wed, 13 May 2026 06:54:01 GMT
• Title: Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
• Abstract summary: We show that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest.<n>We recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor.
• Score: 15.754779716816174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Related papers

• The Spectral Dynamics and Noise Geometry of Muon [0.6335848702857036]
  Muon replaces a matrix gradient $G=UVtop$ by its polar factor $UVtop$.<n>We study the optimization bias created by this operation.
  arXiv  Detail & Related papers  (2026-06-07T00:51:30Z)
• Why Muon Outperforms Adam: A Curvature Perspective [49.85900116602357]
  Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
  arXiv  Detail & Related papers  (2026-06-03T09:40:30Z)
• Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering [32.15111956211244]
  We show that momentum in Muon acts as a spectral filter.<n>Applying momentum before perturbation provably stronger alignment with the signal component of gradient than either reversing this order or simply removing momentum.
  arXiv  Detail & Related papers  (2026-06-02T16:54:38Z)
• DynMuon: A Dynamic Spectral Shaping View of Muon [12.175075916077702]
  We consider a class of Muon-like updates, where we replace the update $M$ with $Up Vtop$ for some parameter $p$.<n>We propose DynMuon, an efficient dynamic spectral shaping method that schedules $p$ from positive to mildly negative over training.
  arXiv  Detail & Related papers  (2026-05-16T18:30:11Z)
• Muon is Not That Special: Random or Inverted Spectra Work Just as Well [50.969177887027115]
  We demonstrate that precise geometric structure is not key factor affecting performance.<n>We introduce Freon, a family of geometrics based on Schatten (quasi-size)norms.<n>We also introduce Kaon, an absurd that replaces singular values with random noise.
  arXiv  Detail & Related papers  (2026-05-11T19:42:48Z)
• Muown: Row-Norm Control for Muon Optimization [41.541871767555484]
  Muon is a strong competitor to AdamW for language model pre-training.<n>Muown improves perplexity over Muon, SOAP, AdamW, and Lion.
  arXiv  Detail & Related papers  (2026-05-11T16:26:06Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory [77.27772368491698]
  Spectrals such as Muon have recently shown strong empirical performance in large-scale language model training.<n>We study this question through the linear associative memory problem.<n>We show that the storage capacity of Muon significantly exceeds that of SGD.
  arXiv  Detail & Related papers  (2026-03-27T16:13:18Z)
• Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning [72.62839712454196]
  Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improveimation.<n>We argue that this "egalitarian" constraint is suboptimal for Deep Neural Networks, where the curvature spectrum is known to be highly heavy-tailed and ill-conditioned.<n>We propose textbfMousse (textbfMuon textbfOptimization textbfUtilizing textbfShampoo's textbfStruct
  arXiv  Detail & Related papers  (2026-03-10T14:03:49Z)
• Muon in Associative Memory Learning: Training Dynamics and Scaling Laws [23.350512542598803]
  We study Muon in a linear associative memory model with softmax retrieval and a hierarchical frequency spectrum over query-answer pairs.<n>We show that Muon mitigates this imbalance, leading to faster and more uniform progress.
  arXiv  Detail & Related papers  (2026-02-05T14:49:40Z)
• Delving into Muon and Beyond: Deep Analysis and Extensions [8.297062899157664]
  We study Muon as the p = 0 endpoint of a family of spectral transformations of the form U boldsymbolp V'.<n>We find that RMS-normalized updates yield more stable optimization than first-moment updates.<n>Our results suggest that Muon is best understood as an effective form of spectral normalization, but not a universally superior optimization method.
  arXiv  Detail & Related papers  (2026-02-04T15:40:47Z)
• Preconditioning Benefits of Spectral Orthogonalization in Muon [50.62925024212989]
  We study the effectiveness of a simplified variant of Muon in two case studies: matrix factorization and in-context learning of linear transformers.<n>Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior.
  arXiv  Detail & Related papers  (2026-01-20T00:08:31Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)
• Convergence of Batch Stochastic Gradient Descent Methods with Approximate Gradients and/or Noisy Measurements: Theory and Computational Results [0.9900482274337404]
  We study convex optimization using a very general formulation called BSGD (Block Gradient Descent) We establish conditions for BSGD to converge to the global minimum, based on approximation theory. We show that when approximate gradients are used, BSGD converges while momentum-based methods can diverge.
  arXiv  Detail & Related papers  (2022-09-12T16:23:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.