Insights on Muon from Simple Quadratics

• URL: http://arxiv.org/abs/2602.11948v1
• Date: Thu, 12 Feb 2026 13:43:58 GMT
• Title: Insights on Muon from Simple Quadratics
• Abstract summary: Muon updates weight matrices along (approximate) polar factors of the gradients.<n>Existing attempts at explaining its performance largely focus on single-step comparisons.<n>We show that understanding Muon requires going beyond local proxies and pessimistic worst-case bounds.
• Score: 2.8348950186890467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon updates weight matrices along (approximate) polar factors of the gradients and has shown strong empirical performance in large-scale training. Existing attempts at explaining its performance largely focus on single-step comparisons (on quadratic proxies) and worst-case guarantees that treat the inexactness of the polar-factor as a nuisance ``to be argued away''. We show that already on simple strongly convex functions such as $L(W)=\frac12\|W\|_{\text{F}}^2$, these perspectives are insufficient, suggesting that understanding Muon requires going beyond local proxies and pessimistic worst-case bounds. Instead, our analysis exposes two observations that already affect behavior on simple quadratics and are not well captured by prevailing abstractions: (i) approximation error in the polar step can qualitatively alter discrete-time dynamics and improve reachability and finite-time performance -- an effect practitioners exploit to tune Muon, but that existing theory largely treats as a pure accuracy compromise; and (ii) structural properties of the objective affect finite-budget constants beyond the prevailing conditioning-based explanations. Thus, any general theory covering these cases must either incorporate these ingredients explicitly or explain why they are irrelevant in the regimes of interest.

Related papers

• Variational Inference for Evidential Deep Learning [69.9353638875714]
  We propose a mathematically principled framework, Variational Inference Evidential Deep Learning (VI-EDL)<n>VI-EDL achieves state-of-the-art performance in out-of-distribution detection, noise detection and autonomous driving scenario.
  arXiv  Detail & Related papers  (2026-05-26T02:32:59Z)
• Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise [43.39716211464324]
  Sign-based optimization algorithms such as Lion and Muon have recently demonstrated superior empirical performance over AdamW.<n>In this paper, we aim to bridge the gap between theory and practice through the lens of heavy-tailed gradient noise.
  arXiv  Detail & Related papers  (2026-02-07T07:47:14Z)
• Preconditioning Benefits of Spectral Orthogonalization in Muon [50.62925024212989]
  We study the effectiveness of a simplified variant of Muon in two case studies: matrix factorization and in-context learning of linear transformers.<n>Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior.
  arXiv  Detail & Related papers  (2026-01-20T00:08:31Z)
• On the Sample Complexity of Learning for Blind Inverse Problems [8.091533689145908]
  Blind inverse problems arise in many experimental settings where the forward operator is partially or entirely unknown.<n>Data-driven approaches have been proposed to address blind inverse problems, demonstrating strong empirical performance and adaptability.<n>We shed light on learning in blind inverse problems within the simplified yet insightful framework of Linear Minimum Mean Square Estimators.
  arXiv  Detail & Related papers  (2025-12-29T11:53:37Z)
• Convergence Bound and Critical Batch Size of Muon Optimizer [1.2289361708127877]
  We provide convergence proofs for Muon across four practical settings.<n>We show that the addition of weight decay yields strictly tighter theoretical bounds.<n>We derive the critical batch size for Muon that minimizes the computational cost of training.
  arXiv  Detail & Related papers  (2025-07-02T11:03:13Z)
• Investigating the Impact of Model Instability on Explanations and Uncertainty [43.254616360807496]
  We simulate uncertainty in text input by introducing noise at inference time. We find that high uncertainty doesn't necessarily imply low explanation plausibility. This suggests that noise-augmented models may be better at identifying salient tokens when uncertain.
  arXiv  Detail & Related papers  (2024-02-20T13:41:21Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• Causal Expectation-Maximisation [70.45873402967297]
  We show that causal inference is NP-hard even in models characterised by polytree-shaped graphs. We introduce the causal EM algorithm to reconstruct the uncertainty about the latent variables from data about categorical manifest variables. We argue that there appears to be an unnoticed limitation to the trending idea that counterfactual bounds can often be computed without knowledge of the structural equations.
  arXiv  Detail & Related papers  (2020-11-04T10:25:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.