Muon is Not That Special: Random or Inverted Spectra Work Just as Well

• URL: http://arxiv.org/abs/2605.11181v1
• Date: Mon, 11 May 2026 19:42:48 GMT
• Title: Muon is Not That Special: Random or Inverted Spectra Work Just as Well
• Abstract summary: We demonstrate that precise geometric structure is not key factor affecting performance.<n>We introduce Freon, a family of geometrics based on Schatten (quasi-size)norms.<n>We also introduce Kaon, an absurd that replaces singular values with random noise.
• Score: 50.969177887027115
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear minimization oracle (LMO) theory. In this paper, we challenge this geometric narrative through three contributions, demonstrating that precise geometric structure is not the key factor affecting optimization performance. First, we introduce Freon, a family of optimizers based on Schatten (quasi-)norms, powered by a novel, provably optimal QDWH-based iterative approximation. Freon naturally interpolates between SGD and Muon, while smoothly extrapolating into the quasi-norm regime. Empirically, the best-performing Schatten parameters for GPT-2 lie strictly within the quasi-norm regime, and thus cannot be represented by any unitarily invariant LMO. Second, noting that Freon performs well across a wide range of exponents, we introduce Kaon, an absurd optimizer that replaces singular values with random noise. Despite lacking any coherent geometric structure, Kaon matches Muon's performance and retains classical convergence guarantees, proving that strict adherence to a precise geometry is practically irrelevant. Third, having shown that geometry is not the primary driver of performance, we demonstrate it is instead controlled by two local quantities: alignment and descent potential. Ultimately, each optimizer must tune its step size around these two quantities. While their dynamics are difficult to predict a-priori, evaluating them within a stochastic random feature model yields a precise insight: Muon succeeds not by tracking an ideal global geometry, but by guaranteeing step-size optimality.

Related papers

• Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization [12.70261661948628]
  We show that finite Newton-Schulz iteration can be beneficial for nonsmooth non optimization.<n>The argument extends beyond Newton-Schulz to general spectral maps of the same property.
  arXiv  Detail & Related papers  (2026-08-26T18:14:00Z)
• OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality [23.28384210732827]
  We show that closed-loop scalar momentum.<n>optimization can be combined with Muon-style momentum.<n>optimization while retaining noise adaptivity and zero-noise optimality up to logarithmic factors.<n>These results show that OptMuon-A achieves the noise-rate.<n>(tildemathcal O(T-1/2+1/2T--1/2)) under average smoothness, while OptMuon-I achieves the noise-rate.<n>(tildemathcal O(T-1/2+
  arXiv  Detail & Related papers  (2026-06-07T18:59:24Z)
• Fitting Unknown Number of Hyperplanes with Manifold Optimization [57.48093263119306]
  Fitting unknown number of linearplanes to data is a fundamental yet challenging problem in machine learning.<n>Existing approaches often struggle with optimum optima or lack geometric consistency.
  arXiv  Detail & Related papers  (2026-05-27T14:02:20Z)
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms [3.5975968496682484]
  Moderns like Muon impose matrix-wise geometry constraints on their updates.<n>All current methods impose fixed LMO geometries for the update rules, chosen by-design or empirically, which are not optimal according to the problem's geometry.<n>We introduce a novel efficient datadriven criterion for dynamically choosing proxy-optimal update LMO geometries.
  arXiv  Detail & Related papers  (2026-05-19T12:47:41Z)
• Phases of Muon: When Muon Eclipses SignSGD [15.75466352147896]
  Muon and related spectrals have demonstrated strong empirical performance as scalable methods, often outperforming Adam.<n>We analyze spectrals, including Muon, on a high-dimensional matrix-valued least squares problem.<n>We derive explicit deterministic dynamics that provide a tractable framework for studying learning behaviour with a focus on (stochastic) SignSVD, which Muon approximates, and (stochastic) SignSGD, the latter serving as a proxy for Adam.
  arXiv  Detail & Related papers  (2026-05-10T14:11:22Z)
• Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer [52.9566744833465]
  Matrix-based approaches to training Large Language Models (LLMs) have demonstrated immense potential.<n>A superior must satisfy three core desiderata: efficiency, achieving Muon-like preconditioning to accelerate optimization; stability, strictly adhering to the inherent in neural networks; and speed, minimizing computational overhead.<n>We propose Nora, which rigorously satisfies all three requirements and is highly promising for large-scale training.
  arXiv  Detail & Related papers  (2026-05-05T14:00:27Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Delving into Muon and Beyond: Deep Analysis and Extensions [8.297062899157664]
  We study Muon as the p = 0 endpoint of a family of spectral transformations of the form U boldsymbolp V'.<n>We find that RMS-normalized updates yield more stable optimization than first-moment updates.<n>Our results suggest that Muon is best understood as an effective form of spectral normalization, but not a universally superior optimization method.
  arXiv  Detail & Related papers  (2026-02-04T15:40:47Z)
• Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods [50.070182958880146]
  We propose a unified framework generalizing descent, quasi-Newton methods, and adaptive methods through the novel notion of preconditioned matrix norms.<n>Within this framework, we provide the first systematic treatment of affine and scale invariance in the matrix- parameterized setting.<n>We introduce two new methods, $ttMuAdam$ and $texttMuAdam-SANIA$, which combine the spectral geometry of Muon with Adam-style preconditioning.
  arXiv  Detail & Related papers  (2025-10-12T19:39:41Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Error Feedback for Muon and Friends [80.90330715662961]
  We introduce EF21-Muon, the first communication-efficient, non-Euclidean LMO-based with rigorous convergence guarantees.<n>Our theory covers non-Euclidean smooth and the more general $(L0, L1)$-smooth setting, matching best-known Euclidean rates and enabling faster convergence under suitable norm choices.
  arXiv  Detail & Related papers  (2025-10-01T08:20:08Z)
• On the Convergence of Muon and Beyond [31.900178928104648]
  We provide the first proof that variance reduction enables Muon-MVR2 to attain the optimal complexity.<n>Overall, this work offers the first proof of optimality for a Muon-style.
  arXiv  Detail & Related papers  (2025-09-19T09:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.