The Spectral Dynamics and Noise Geometry of Muon

• URL: http://arxiv.org/abs/2606.08388v1
• Date: Sun, 07 Jun 2026 00:51:30 GMT
• Title: The Spectral Dynamics and Noise Geometry of Muon
• Abstract summary: Muon replaces a matrix gradient $G=UVtop$ by its polar factor $UVtop$.<n>We study the optimization bias created by this operation.
• Score: 0.6335848702857036
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon replaces a matrix gradient $G=UΣV^\top$ by its polar factor $UV^\top$. This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We study the optimization bias created by this operation. Under explicit alignment assumptions, we prove that the polar update is the one-step entropy-maximizing choice among bounded updates that use the gradient singular directions and do not adapt to the current weight spectrum. In an underdetermined regression model, we derive exact singular-value dynamics for continuous-time Muon and identify a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. This geometry also rules out a common low-rank interpretation: at fixed Frobenius norm, Muon's distinguished state has a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments separate the effect from simple gradient rescaling, show that norm-matched gradient descent does not reproduce Muon, and recover the predicted flattening trend across broad ablations. In small NanoGPT pretraining, Muon preserves stable rank, has a broad learning-rate plateau, and improves validation loss relative to AdamW; in a matched small-ViT control, the ranking reverses. The resulting picture is regime-dependent: Muon is not universally superior, but its flat-spectrum bias can help when many spectral directions need to remain active.

Related papers

• Beyond the Matrix Sign: Quadratic Spectral Descent [6.589542614722917]
  We show that curvature can change both the singular values and the singular directions of the optimal update.<n>Experiments on GPT pre-training show that QSD consistently improves validation loss over Muon and recent Muon variants.
  arXiv  Detail & Related papers  (2026-09-07T15:07:24Z)
• Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent [11.327284468812948]
  We show that negative-shifted gradient descent escapes a structural constraint.<n>The filter is smooth at the would-be pole and mixed-sign-capable.<n>A finite-grid hold-out inequality transfers the separations to the validation-selected algorithm.
  arXiv  Detail & Related papers  (2026-07-24T16:42:46Z)
• An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies [3.9617623336464636]
  Muon and related matrix-signs are increasingly used to pre-train large language models.<n>This report proposes a unified framework built on a single idealizing assumption.<n>We study three systems trained with Muon: a nanoGPT feed-forward projection, a 64-expert mixture-of-experts router, and the query/key projections of a bf16 FlashAttention block.
  arXiv  Detail & Related papers  (2026-07-22T05:39:00Z)
• OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality [23.28384210732827]
  We show that closed-loop scalar momentum.<n>optimization can be combined with Muon-style momentum.<n>optimization while retaining noise adaptivity and zero-noise optimality up to logarithmic factors.<n>These results show that OptMuon-A achieves the noise-rate.<n>(tildemathcal O(T-1/2+1/2T--1/2)) under average smoothness, while OptMuon-I achieves the noise-rate.<n>(tildemathcal O(T-1/2+
  arXiv  Detail & Related papers  (2026-06-07T18:59:24Z)
• Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence [15.754779716816174]
  We show that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest.<n>We recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor.
  arXiv  Detail & Related papers  (2026-05-13T06:54:01Z)
• Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters [48.3175117923623]
  Most gradient clipping rules treat all parameters as heavy vectors and ignore the matrix structure of modern vectors.<n>We show empirically that data outliers often amplify only a small number of leading singular values.<n>Motivated by this phenomenon, we propose clipping, which stabilizes by singular values.
  arXiv  Detail & Related papers  (2026-05-12T09:24:59Z)
• Muon is Not That Special: Random or Inverted Spectra Work Just as Well [50.969177887027115]
  We demonstrate that precise geometric structure is not key factor affecting performance.<n>We introduce Freon, a family of geometrics based on Schatten (quasi-size)norms.<n>We also introduce Kaon, an absurd that replaces singular values with random noise.
  arXiv  Detail & Related papers  (2026-05-11T19:42:48Z)
• Muown: Row-Norm Control for Muon Optimization [41.541871767555484]
  Muon is a strong competitor to AdamW for language model pre-training.<n>Muown improves perplexity over Muon, SOAP, AdamW, and Lion.
  arXiv  Detail & Related papers  (2026-05-11T16:26:06Z)
• Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning [72.62839712454196]
  Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improveimation.<n>We argue that this "egalitarian" constraint is suboptimal for Deep Neural Networks, where the curvature spectrum is known to be highly heavy-tailed and ill-conditioned.<n>We propose textbfMousse (textbfMuon textbfOptimization textbfUtilizing textbfShampoo's textbfStruct
  arXiv  Detail & Related papers  (2026-03-10T14:03:49Z)
• Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval [13.218607858857295]
  Spectral gradient methods modify gradient updates by preserving directional information while discarding scale.<n>We investigate the mechanisms underlying these gains through a dynamical analysis of a nonlinear phase retrieval model.
  arXiv  Detail & Related papers  (2026-01-30T07:12:58Z)
• Preconditioning Benefits of Spectral Orthogonalization in Muon [50.62925024212989]
  We study the effectiveness of a simplified variant of Muon in two case studies: matrix factorization and in-context learning of linear transformers.<n>Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior.
  arXiv  Detail & Related papers  (2026-01-20T00:08:31Z)
• When do spectral gradient updates help in deep learning? [7.5757345574662205]
  We propose a simple condition that predicts when a spectral update yields a larger decrease in the loss than a Euclidean gradient step.<n>We validate these predictions in synthetic regression experiments and in NanoGPT-scale language model training.
  arXiv  Detail & Related papers  (2025-12-03T22:22:09Z)
• Gradient Equilibrium in Online Learning: Theory and Applications [56.02856551198923]
  gradient equilibrium is achieved by standard online learning methods.<n> gradient equilibrium translates into an interpretable and meaningful property in online prediction problems.<n>We show that gradient equilibrium framework can be used to develop a debiasing scheme for black-box predictions.
  arXiv  Detail & Related papers  (2025-01-14T18:59:09Z)
• The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks [117.93273337740442]
  We show that gradient descent converges to a uniform margin classifier on the training data with an $exp(-Omega(log2 t))$ convergence rate. We also show that batch normalization has an implicit bias towards a patch-wise uniform margin.
  arXiv  Detail & Related papers  (2023-06-20T16:58:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.