Beyond the Matrix Sign: Quadratic Spectral Descent

• URL: http://arxiv.org/abs/2609.07597v1
• Date: Mon, 07 Sep 2026 15:07:24 GMT
• Title: Beyond the Matrix Sign: Quadratic Spectral Descent
• Abstract summary: We show that curvature can change both the singular values and the singular directions of the optimal update.<n>Experiments on GPT pre-training show that QSD consistently improves validation loss over Muon and recent Muon variants.
• Score: 6.589542614722917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon can be interpreted as optimizing a linear local objective over a spectral-norm ball. This gives a matrix-sign update that preserves the singular directions of the gradient and assigns the same magnitude to all active singular modes. We ask whether these two properties remain optimal when local curvature is taken into account. To answer this question, we keep Muon's spectral-norm constraint unchanged and replace the linear local model with a quadratic one. We call the resulting method \emph{Quadratic Spectral Descent} (QSD). We show that curvature can change both the singular values and the singular directions of the optimal update. To make QSD practical, we approximate curvature with Kronecker-factored statistics and solve the constrained quadratic with a small number of Frank--Wolfe steps, each of which has a closed-form matrix-sign subproblem. We further provide an optimality certificate, a comparison with Muon under the same quadratic surrogate, and an $O(1/K)$ convergence rate for the inner solver. Experiments on GPT pre-training show that QSD consistently improves validation loss over Muon and recent Muon variants, and reduces wall-clock training time by up to $8.49\%$ at matched validation loss.

Related papers

• Nonsmooth Optimization via Orthogonalized Momentum [11.546795991978613]
  We show that for every fixed momentum factor $in[0,1)$, Muon can fail to approach the global optimal solution a convex Lipschitz objective.<n>We propose MAGD, which combines momentum with gradient, weighted based on their relative progress.
  arXiv  Detail & Related papers  (2026-09-12T03:08:36Z)
• MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning [14.373662883104808]
  We propose MALT to reduce the sensitivity of Muon to curvature geometry of the loss landscape.<n>MALT uses two-sided diagonal preconditioners with low memory and overhead to capture the curvature geometry of the loss landscape.
  arXiv  Detail & Related papers  (2026-08-05T17:26:47Z)
• Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics [0.9798289347944394]
  Matrix factorization is a problem of the form $min_mathbfP,mathbfQ |mathbfMstar - mathbfPtopmathbfQ|_mathrmF2$.<n>We study how trajectories under the Muon parameter differ from those of gradient descent.
  arXiv  Detail & Related papers  (2026-06-29T16:17:38Z)
• Muon$^p$: Muon with Fractional Spectral Powers [3.223825674658464]
  Muon is an increasingly widely used gradient that replaces a $G=USVtop$ with its polar factor $UVtop$.<n>We introduce Muon$p$ that instead uses fractional spectral-power updates $USpVtop$ for rational $pin(0,1)$.<n>Our results reveal important insights on when preserving the singular spectrum can bring significant gains, and introduce a principled way to achieve them.
  arXiv  Detail & Related papers  (2026-06-11T19:48:12Z)
• The Spectral Dynamics and Noise Geometry of Muon [0.6335848702857036]
  Muon replaces a matrix gradient $G=UVtop$ by its polar factor $UVtop$.<n>We study the optimization bias created by this operation.
  arXiv  Detail & Related papers  (2026-06-07T00:51:30Z)
• Muon is Not That Special: Random or Inverted Spectra Work Just as Well [50.969177887027115]
  We demonstrate that precise geometric structure is not key factor affecting performance.<n>We introduce Freon, a family of geometrics based on Schatten (quasi-size)norms.<n>We also introduce Kaon, an absurd that replaces singular values with random noise.
  arXiv  Detail & Related papers  (2026-05-11T19:42:48Z)
• Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization [12.526290684448425]
  Low-rank gradient compression reduces communication in distributed training by representing updates with rank-$r$ factors.<n>We propose Orth-Dion, which replaces column normalization with QR orthogonalization of the right factor.<n>Experiments on large-scale language model pre-training validate the predicted $sqrtr$ scaling and show that Orth-Dion closes the convergence gap to Muon at Dion's communication cost.
  arXiv  Detail & Related papers  (2026-05-07T23:37:37Z)
• Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning [72.62839712454196]
  Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improveimation.<n>We argue that this "egalitarian" constraint is suboptimal for Deep Neural Networks, where the curvature spectrum is known to be highly heavy-tailed and ill-conditioned.<n>We propose textbfMousse (textbfMuon textbfOptimization textbfUtilizing textbfShampoo's textbfStruct
  arXiv  Detail & Related papers  (2026-03-10T14:03:49Z)
• Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimization [54.723834588133165]
  We study online linear optimization with matrix variables by the operatorAML, a setting where the geometry renders designing datadependent and efficient adaptive algorithms challenging.<n>We instantiate this framework with two efficient methods that avoid projections.<n>We show both methods admit closed-form updates match one-sided Shampoo's regret up to a constant factor, while significantly reducing computational cost.
  arXiv  Detail & Related papers  (2026-02-09T03:09:47Z)
• Delving into Muon and Beyond: Deep Analysis and Extensions [8.297062899157664]
  We study Muon as the p = 0 endpoint of a family of spectral transformations of the form U boldsymbolp V'.<n>We find that RMS-normalized updates yield more stable optimization than first-moment updates.<n>Our results suggest that Muon is best understood as an effective form of spectral normalization, but not a universally superior optimization method.
  arXiv  Detail & Related papers  (2026-02-04T15:40:47Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Statistical Inference of Constrained Stochastic Optimization via Sketched Sequential Quadratic Programming [53.63469275932989]
  We consider online statistical inference of constrained nonlinear optimization problems.<n>We apply the Sequential Quadratic Programming (StoSQP) method to solve these problems.
  arXiv  Detail & Related papers  (2022-05-27T00:34:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.