Muon Learns More Robust and Transferable Features than Adam

• URL: http://arxiv.org/abs/2606.09658v1
• Date: Mon, 08 Jun 2026 15:42:54 GMT
• Title: Muon Learns More Robust and Transferable Features than Adam
• Abstract summary: We show that features learned by Muon are consistently more robust than those learned by Adam and SGD.<n>We also demonstrate that Muon-learned features transfer more effectively than those learned by Adam and SGD.
• Score: 24.749527924478148
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon has recently emerged as a state-of-the-art optimizer for pretraining Large Language Models (LLMs) and vision classifiers. Despite its efficiency advantage over Adam and SGD, the feature-learning advantage of Muon remains unclear. This paper investigates Muon's feature-learning advantage through the lens of robustness and transferability. First, by evaluating pretrained models on corrupted images and texts, we show that features learned by Muon are consistently more robust than those learned by Adam and SGD across different architectures, including transformers and Convolutional Neural Networks (CNNs). Using trained layer-wise probes, we further show that this robustness advantage is reflected in larger logit margins across layers. Second, by training linear classifiers or fine-tuning full models from pretrained parameters on downstream tasks, we demonstrate that Muon-learned features transfer more effectively than those learned by Adam and SGD. This transferability advantage is further supported by the diversity of hidden states across layers, as measured by effective rank. Finally, in a representative classification problem with multi-component features, we prove that Muon attains larger margins and higher effective rank than Adam and SGD, providing theoretical support for our empirical findings.

Related papers

• Reassessing Muon for Matrix Factorization [9.691906492780353]
  Muon has emerged as a strong contender in large-scale deep learning.<n>We study Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization.<n>We find that Muon does not consistently outperform AdamW in this setting.
  arXiv  Detail & Related papers  (2026-07-14T20:21:38Z)
• Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials [0.5714512533033075]
  We implement and compare a class of recently proposed matrix-structureds, including Muon, SOAP, and the hybrid SOAP-Muon, for training NequIP and Allegro MLIP models.<n>We find that theses can substantially outperform Adam in both convergence speed and final accuracy.
  arXiv  Detail & Related papers  (2026-07-02T17:57:31Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Muon Outperforms Adam in Tail-End Associative Memory Learning [118.98991042050532]
  We show that Muon consistently achieves balanced learning across classes regardless of feature embeddings.<n>Our empirical observations and theoretical analyses reveal Muon's core advantage: its update rule aligns with the outer-product structure of linear associative memories.
  arXiv  Detail & Related papers  (2025-09-30T10:04:08Z)
• Transferring Linear Features Across Language Models With Model Stitching [65.74996520496195]
  We show that affine mappings between residual streams of language models is a cheap way to transfer represented features between models.<n>We find that small and large models learn similar representation spaces, which motivates training expensive components like SAEs on a smaller model and transferring to a larger model at a FLOPs savings.
  arXiv  Detail & Related papers  (2025-06-07T01:03:25Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)
• MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representations [2.94944680995069]
  We propose a multi-task self-supervised framework (MT-SLVR) that learns both variant and invariant features in a parameter-efficient manner. We evaluate our approach on few-shot classification tasks drawn from a variety of audio domains and demonstrate improved classification performance.
  arXiv  Detail & Related papers  (2023-05-29T09:10:50Z)
• Learning Invariant Representations across Domains and Tasks [81.30046935430791]
  We propose a novel Task Adaptation Network (TAN) to solve this unsupervised task transfer problem. In addition to learning transferable features via domain-adversarial training, we propose a novel task semantic adaptor that uses the learning-to-learn strategy to adapt the task semantics. TAN significantly increases the recall and F1 score by 5.0% and 7.8% compared to recently strong baselines.
  arXiv  Detail & Related papers  (2021-03-03T11:18:43Z)
• Do Adversarially Robust ImageNet Models Transfer Better? [102.09335596483695]
  adversarially robust models often perform better than their standard-trained counterparts when used for transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations.
  arXiv  Detail & Related papers  (2020-07-16T17:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.