Muon as a Residual Connection

• URL: http://arxiv.org/abs/2607.01124v1
• Date: Wed, 01 Jul 2026 16:12:24 GMT
• Title: Muon as a Residual Connection
• Abstract summary: Muon has emerged as one of the most effective preservations for training large neural networks.<n>We propose a mechanistic interpretation: Muon can be understood as an implicit residual connection during training.
• Score: 4.242184336943779
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon has recently emerged as one of the most effective optimizers for training large neural networks, yet its empirical success has been explained from several different perspectives. In this paper, we propose a simple mechanistic interpretation: Muon can be understood as an implicit residual connection during training. Specifically, orthogonalizing the update can sacrifice some immediate gradient fidelity while improving representation preservation for downstream layers. We study this trade-off in controlled linear optimization settings, where Muon can learn representations that are slower to fit a local target but easier for downstream layers to exploit. Our results suggest a conceptual explanation for Muon and a design perspective for optimizers that balance local descent with downstream usability.

Related papers

• Reassessing Muon for Matrix Factorization [9.691906492780353]
  Muon has emerged as a strong contender in large-scale deep learning.<n>We study Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization.<n>We find that Muon does not consistently outperform AdamW in this setting.
  arXiv  Detail & Related papers  (2026-07-14T20:21:38Z)
• Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective [12.959531584997462]
  Recently, Muon has gained substantial attention as an appealing alternative to Adam-like nuisances.<n>We introduce a mixed-spiked matrix sensing model whose sensing operator decomposes into signal, spike, and bulk components.<n>We show that Muon moves faster along the information-bearing river direction during early stages of learning.
  arXiv  Detail & Related papers  (2026-06-19T15:10:20Z)
• Muon Learns More Robust and Transferable Features than Adam [24.749527924478148]
  We show that features learned by Muon are consistently more robust than those learned by Adam and SGD.<n>We also demonstrate that Muon-learned features transfer more effectively than those learned by Adam and SGD.
  arXiv  Detail & Related papers  (2026-06-08T15:42:54Z)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory [77.27772368491698]
  Spectrals such as Muon have recently shown strong empirical performance in large-scale language model training.<n>We study this question through the linear associative memory problem.<n>We show that the storage capacity of Muon significantly exceeds that of SGD.
  arXiv  Detail & Related papers  (2026-03-27T16:13:18Z)
• NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training [50.27276603708547]
  We show that despite imposing full-rank updates, Muon-trained models exhibit pronounced low-rank structure in their weight matrices and are readily compressible under standard pipelines.<n>We propose NuMuon, which augments Muon with a nuclear-norm constraint on the update direction, further constraining the learned weights toward low-rank structure.
  arXiv  Detail & Related papers  (2026-03-04T00:10:14Z)
• To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters [16.624341041698013]
  Muon has perhaps gained the highest popularity due to its superior training speed.<n>This paper investigates the potential downsides stemming from the mechanism driving this speedup.<n>Muon struggles to uncover common underlying structure across tasks, and is more prone to fitting spurious features.
  arXiv  Detail & Related papers  (2026-02-28T17:37:15Z)
• Muon in Associative Memory Learning: Training Dynamics and Scaling Laws [23.350512542598803]
  We study Muon in a linear associative memory model with softmax retrieval and a hierarchical frequency spectrum over query-answer pairs.<n>We show that Muon mitigates this imbalance, leading to faster and more uniform progress.
  arXiv  Detail & Related papers  (2026-02-05T14:49:40Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• FedMuon: Federated Learning with Bias-corrected LMO-based Optimization [36.00641661700195]
  We study how Muon can be utilized in federated learning.<n>We demonstrate that FedMuon can outperform the state-of-the-art federated learning methods.
  arXiv  Detail & Related papers  (2025-09-30T14:45:12Z)
• Muon Outperforms Adam in Tail-End Associative Memory Learning [118.98991042050532]
  We show that Muon consistently achieves balanced learning across classes regardless of feature embeddings.<n>Our empirical observations and theoretical analyses reveal Muon's core advantage: its update rule aligns with the outer-product structure of linear associative memories.
  arXiv  Detail & Related papers  (2025-09-30T10:04:08Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)
• Improving Network Interpretability via Explanation Consistency Evaluation [56.14036428778861]
  We propose a framework that acquires more explainable activation heatmaps and simultaneously increase the model performance. Specifically, our framework introduces a new metric, i.e., explanation consistency, to reweight the training samples adaptively in model learning. Our framework then promotes the model learning by paying closer attention to those training samples with a high difference in explanations.
  arXiv  Detail & Related papers  (2024-08-08T17:20:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.