An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies

• URL: http://arxiv.org/abs/2607.19771v1
• Date: Wed, 22 Jul 2026 05:39:00 GMT
• Title: An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies
• Abstract summary: Muon and related matrix-signs are increasingly used to pre-train large language models.<n>This report proposes a unified framework built on a single idealizing assumption.<n>We study three systems trained with Muon: a nanoGPT feed-forward projection, a 64-expert mixture-of-experts router, and the query/key projections of a bf16 FlashAttention block.
• Score: 3.9617623336464636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon and related matrix-sign optimizers are increasingly used to pre-train large language models, but their effect on the internal geometry of individual weight matrices is not well understood. This preliminary report proposes a unified framework built on a single idealizing assumption -- exact scale invariance of the loss under weight rescaling, which holds approximately in normalization-heavy networks. Under this assumption, plain SGD carries a built-in 1/||W|| brake on its update size, whereas Muon's matrix-sign step removes that brake, so both the Frobenius and spectral norms drift outward faster (t^{1/2} versus t^{1/4}). We further observe that the spectral-norm perturbation has a non-negative second-order term. This implies that a lightweight "spectral cap" -- which projects out only the first-order growth of the single top singular direction from each update -- can control the output covariance W K_X W^T without freezing training: the weight keeps learning through non-top directions, top-direction rotation, and top switching. We relate this cap to the min-entropy (H-infinity) of the singular-value spectrum. We then study three systems trained with Muon: a nanoGPT feed-forward projection, a 64-expert mixture-of-experts router, and the query/key projections of a bf16 FlashAttention block. In each case the cap increases isotropy and, at the margins -- a router collapsing to a single expert, and the near-divergence of one attention head -- prevents a concrete failure, while leaving validation loss essentially unchanged. We emphasize that the scale-invariance assumption is strong and that these small-scale results are preliminary; comments are welcome.

Related papers

• Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent [11.327284468812948]
  We show that negative-shifted gradient descent escapes a structural constraint.<n>The filter is smooth at the would-be pole and mixed-sign-capable.<n>A finite-grid hold-out inequality transfers the separations to the validation-selected algorithm.
  arXiv  Detail & Related papers  (2026-07-24T16:42:46Z)
• The Spectral Dynamics and Noise Geometry of Muon [0.6335848702857036]
  Muon replaces a matrix gradient $G=UVtop$ by its polar factor $UVtop$.<n>We study the optimization bias created by this operation.
  arXiv  Detail & Related papers  (2026-06-07T00:51:30Z)
• Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training [56.323119575322146]
  Extra-Merge consistently outperforms standard merging baselines in experiments across GPT-2 and LLaMA families.<n>It yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon citepjordan2024muon.
  arXiv  Detail & Related papers  (2026-05-26T02:48:34Z)
• From Latent Space to Training Data: Explainable Specialization in Minimal MLPs [0.47554242259005197]
  We study whether training biases can make hidden neurons specialize in minimal one-hidden-layers.<n>Experiments on uniformly sampled one-dimensional datasets show a stable pattern from N = 3 to N = 100 across 480 controlled runs.
  arXiv  Detail & Related papers  (2026-05-25T15:18:22Z)
• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• Muown: Row-Norm Control for Muon Optimization [41.541871767555484]
  Muon is a strong competitor to AdamW for language model pre-training.<n>Muown improves perplexity over Muon, SOAP, AdamW, and Lion.
  arXiv  Detail & Related papers  (2026-05-11T16:26:06Z)
• A Nonasymptotic Theory of Gain-Dependent Error Dynamics in Behavior Cloning [4.822598110892847]
  Behavior cloning policies on position-controlled robots inherit the closed-loop response of the underlying PD controller.<n>We show that independent sub-Gaussian action errors propagate through the gain-dependent closed-loop dynamics.
  arXiv  Detail & Related papers  (2026-04-15T23:53:09Z)
• Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold [0.0]
  Generalization requires the textitanisotropic rectification unique to adaptives, which directs noise into the tangent space of the solution.<n>This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a Spectral Gating'' mechanism that regulates the transition from generalization to memorization.
  arXiv  Detail & Related papers  (2026-03-16T16:18:18Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Deriving Decoder-Free Sparse Autoencoders from First Principles [0.0]
  Theory predicts collapse without volume control analogous to the log-determinant in Gaussian mixture models.<n>We instantiate the theory in a single-layer encoder with an LSE objective and InfoMax regularization for volume control.<n>Experiments confirm the theory's predictions.
  arXiv  Detail & Related papers  (2026-01-10T08:13:58Z)
• Error Feedback for Muon and Friends [80.90330715662961]
  We introduce EF21-Muon, the first communication-efficient, non-Euclidean LMO-based with rigorous convergence guarantees.<n>Our theory covers non-Euclidean smooth and the more general $(L0, L1)$-smooth setting, matching best-known Euclidean rates and enabling faster convergence under suitable norm choices.
  arXiv  Detail & Related papers  (2025-10-01T08:20:08Z)
• Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion [55.95767828747407]
  In domains such as molecular and protein generation, physical systems exhibit inherent symmetries that are critical to model.<n>We present a framework that reduces training variance and provides a provably lower-variance gradient estimator.<n>We also present a practical implementation of this estimator incorporating the loss and sampling procedure through a method we call Orbit Diffusion.
  arXiv  Detail & Related papers  (2025-02-14T03:26:57Z)
• Debiased Sinkhorn barycenters [110.79706180350507]
  Entropy regularization in optimal transport (OT) has been the driver of many recent interests for Wasserstein metrics and barycenters in machine learning. We show how this bias is tightly linked to the reference measure that defines the entropy regularizer. We propose debiased Wasserstein barycenters that preserve the best of both worlds: fast Sinkhorn-like iterations without entropy smoothing.
  arXiv  Detail & Related papers  (2020-06-03T23:06:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.