Related papers: Can Muon Fine-tune Adam-Pretrained Models?

Can Muon Fine-tune Adam-Pretrained Models?

URL: http://arxiv.org/abs/2605.10468v1
Date: Mon, 11 May 2026 12:34:20 GMT
Title: Can Muon Fine-tune Adam-Pretrained Models?
Abstract summary: Most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to a mismatch.<n>We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength.<n>We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning.
Score: 0.5735035463793009
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to an optimizer mismatch. We investigate this mismatch through controlled experiments and relate it to the distinct implicit biases of Adam and Muon. We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength. This leads us to hypothesize that constraining updates should mitigate the mismatch. We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning. Studies on LoRA rank, catastrophic forgetting, and LoRA variants further confirm that mismatch severity correlates with update strength. These results shed light on how optimizer mismatch affects fine-tuning and how it can be mitigated. Our code is available at https://github.com/XingyuQu/muon-finetune.

Related papers

When Does Muon Help Agentic Reinforcement Learning? [14.63102636181275]
Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning post-training remains unclear.<n>We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld.
arXiv Detail & Related papers (2026-07-17T17:49:05Z)
Open Problem: Is AdamW Effective Under Heavy-Tailed Noise? [43.39716211464324]
AdamW is the de facto theory for training large language models.<n>Recent work shows that sign-based gradients such as Lion and Muon achieve sharp heavy-tailed rates.<n>Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction?
arXiv Detail & Related papers (2026-06-22T17:58:52Z)
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold [13.757351633281836]
Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models.<n>We derive LoRA-Muon by applying the Muon's steepest-descent rule to the low-rank setting.
arXiv Detail & Related papers (2026-06-11T05:24:45Z)
Why Muon Outperforms Adam: A Curvature Perspective [49.85900116602357]
Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
arXiv Detail & Related papers (2026-06-03T09:40:30Z)
LionMuon: Alternating Spectral and Sign Descent for Efficient Training [48.69598464267559]
We propose LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged cost.<n>It alternates between Lion's and Muon's updates on a fixed period P, sharing a single dual-EMA momentum buffer between them.<n>A single-EMA variant, SignMuon, by itself already outperforms pure Muon.
arXiv Detail & Related papers (2026-05-19T13:07:59Z)
Muon Outperforms Adam in Tail-End Associative Memory Learning [118.98991042050532]
We show that Muon consistently achieves balanced learning across classes regardless of feature embeddings.<n>Our empirical observations and theoretical analyses reveal Muon's core advantage: its update rule aligns with the outer-product structure of linear associative memories.
arXiv Detail & Related papers (2025-09-30T10:04:08Z)
Conda: Column-Normalized Adam for Training Large Language Models Faster [70.66067959375748]
Column-Normalized Adam (Conda) is a novel approach to large language models (LLMs)<n>Conda projects updates into a subspace and applies column-wise second moment normalization based on the projected gradients.<n>Experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training.
arXiv Detail & Related papers (2025-09-29T02:58:19Z)
Muon is Scalable for LLM Training [50.68746986439438]
We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
arXiv Detail & Related papers (2025-02-24T09:12:29Z)
Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps [65.64965527170156]
We adapt the widely used Adam optimiser for use in reinforcement learning.<n>We show that Adam-Rel uses the local timestep within an epoch, essentially resetting Adam's timestep to 0 after target changes.<n>We then show that increases in gradient norm occur in RL in practice, and examine the differences between our theoretical model and the observed data.
arXiv Detail & Related papers (2024-12-22T18:01:08Z)
BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models [4.265831047857601]
BAdam is an optimization method that leverages the block coordinate descent (BCD) framework with Adam's update rule. BAdam offers a memory efficient approach to the full parameter finetuning of large language models. Our code can be easily integrated into any PyTorch-based system.
arXiv Detail & Related papers (2024-04-03T15:59:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.