DynMuon: A Dynamic Spectral Shaping View of Muon

• URL: http://arxiv.org/abs/2605.17109v2
• Date: Fri, 22 May 2026 06:10:21 GMT
• Title: DynMuon: A Dynamic Spectral Shaping View of Muon
• Abstract summary: We consider a class of Muon-like updates, where we replace the update $M$ with $Up Vtop$ for some parameter $p$.<n>We propose DynMuon, an efficient dynamic spectral shaping method that schedules $p$ from positive to mildly negative over training.
• Score: 12.175075916077702
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly. The essential difference, when compared to standard gradient descent methods, is to replace the usual update matrix $M=UΣV^\top$ with its polar factor $UV^\top$. In this work, we consider a class of Muon-like updates, where we replace the update $M$ with $UΣ^p V^\top$ for some parameter $p$. We call this a "spectral-shaping" operation, and develop a theory of how to pick $p$ which depends on (a) local curvature of the loss function, (b) noise stemming from stochastic gradients and label noise, and (c) training stage. Our theory and experimentation reveal a previously overlooked behavior: positive $p$ helps early by emphasizing high-curvature directions and accelerating signal contraction, while mildly negative $p$ helps later by reallocating update strength toward low-curvature directions that still contain useful training signals. Building on the insight, we propose DynMuon, an efficient dynamic spectral shaping method that schedules $p$ from positive to mildly negative over training. Extensive experiments across model sizes, architectures, and training settings show that DynMuon consistently achieves lower validation loss than Muon, while requiring 10.6-26.5% fewer steps to reach the same target loss.

Related papers

• Muon$^p$: Muon with Fractional Spectral Powers [3.223825674658464]
  Muon is an increasingly widely used gradient that replaces a $G=USVtop$ with its polar factor $UVtop$.<n>We introduce Muon$p$ that instead uses fractional spectral-power updates $USpVtop$ for rational $pin(0,1)$.<n>Our results reveal important insights on when preserving the singular spectrum can bring significant gains, and introduce a principled way to achieve them.
  arXiv  Detail & Related papers  (2026-06-11T19:48:12Z)
• Why Muon Outperforms Adam: A Curvature Perspective [49.85900116602357]
  Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
  arXiv  Detail & Related papers  (2026-06-03T09:40:30Z)
• Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence [15.754779716816174]
  We show that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest.<n>We recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor.
  arXiv  Detail & Related papers  (2026-05-13T06:54:01Z)
• Muown: Row-Norm Control for Muon Optimization [41.541871767555484]
  Muon is a strong competitor to AdamW for language model pre-training.<n>Muown improves perplexity over Muon, SOAP, AdamW, and Lion.
  arXiv  Detail & Related papers  (2026-05-11T16:26:06Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• The Newton-Muon Optimizer [2.1583596008762935]
  We introduce a surrogate model that sheds new light on the design iteration of Muon.<n>New optimization method, which we refer to as Newton-Muon, shows that standard Muon can be interpreted as an implicit Newton-type method.
  arXiv  Detail & Related papers  (2026-04-01T23:29:08Z)
• Muon in Associative Memory Learning: Training Dynamics and Scaling Laws [23.350512542598803]
  We study Muon in a linear associative memory model with softmax retrieval and a hierarchical frequency spectrum over query-answer pairs.<n>We show that Muon mitigates this imbalance, leading to faster and more uniform progress.
  arXiv  Detail & Related papers  (2026-02-05T14:49:40Z)
• Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards [52.90573877727541]
  reinforcement learning (RL) has been considered for diffusion model fine-tuning.<n>RL's effectiveness is limited by the challenge of sparse reward.<n>$textB2text-DiffuRL$ is compatible with existing optimization algorithms.
  arXiv  Detail & Related papers  (2025-03-14T09:45:19Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)
• High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation [89.21686761957383]
  We study the first gradient descent step on the first-layer parameters $boldsymbolW$ in a two-layer network. Our results demonstrate that even one step can lead to a considerable advantage over random features.
  arXiv  Detail & Related papers  (2022-05-03T12:09:59Z)
• Beyond Lazy Training for Over-parameterized Tensor Decomposition [69.4699995828506]
  We show that gradient descent on over-parametrized objective could go beyond the lazy training regime and utilize certain low-rank structure in the data. Our results show that gradient descent on over-parametrized objective could go beyond the lazy training regime and utilize certain low-rank structure in the data.
  arXiv  Detail & Related papers  (2020-10-22T00:32:12Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.