From SGD to Muon: Adaptive Optimization via Schatten-p Norms

• URL: http://arxiv.org/abs/2605.19781v1
• Date: Tue, 19 May 2026 12:47:41 GMT
• Title: From SGD to Muon: Adaptive Optimization via Schatten-p Norms
• Abstract summary: Moderns like Muon impose matrix-wise geometry constraints on their updates.<n>All current methods impose fixed LMO geometries for the update rules, chosen by-design or empirically, which are not optimal according to the problem's geometry.<n>We introduce a novel efficient datadriven criterion for dynamically choosing proxy-optimal update LMO geometries.
• Score: 3.5975968496682484
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern optimizers, like Muon, impose matrix-wise geometry constraints on their updates. These matrix-wise constraints can be unified under Linear Minimization Oracle (LMO) theory. However, all current methods impose fixed LMO geometries for the update rules, chosen by-design or empirically, which are not necessarily optimal according to the problem's geometry. We introduce a novel efficient datadriven criterion for dynamically choosing proxy-optimal update LMO geometries on individual Deep Neural Network layers. Derived in closed form from gradient and activation statistics using a single-step random feature regression surrogate model, our criterion navigates a design space interpolating from SGD to Muon updates. Moreover, integrating parameter-wise preconditioning allows our framework to recover SGD, Muon, Adam, and MuAdam as specific extrema. To make this adaptive approach scalable, we pair it with efficient computational strategies, achieving only a $\sim$ 3% runtime overhead on highly optimized baselines. As a proof of concept, we show that this data-driven optimizer beats or remains competitive with the performance of the best performing optimizer between Muon and AdamW across three different training scenarios. Ultimately, this work provides evidence that LMO geometry can be successfully and efficiently adapted from runtime data, opening a new pathway for optimizer design beyond static geometries.

Related papers

• Blog: Survey of Optimizers [0.0]
  Survey organizes recent and training optimization methods along four axes: temporal estimation, update geometry, horizon management, and representation and systems.<n>It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficients, schedule-free training, small-batch corrections, and quantized states.
  arXiv  Detail & Related papers  (2026-08-28T17:35:11Z)
• Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning [46.539149169728624]
  A major obstacle in large-scale fine-tuning is the memory overhead of backpropagation.<n>$texttAdaNAGED$ is a method that unifies gradient-free training, adaptive tuning, and non-Euclidean update geometry.
  arXiv  Detail & Related papers  (2026-06-12T21:46:54Z)
• Muon is Not That Special: Random or Inverted Spectra Work Just as Well [50.969177887027115]
  We demonstrate that precise geometric structure is not key factor affecting performance.<n>We introduce Freon, a family of geometrics based on Schatten (quasi-size)norms.<n>We also introduce Kaon, an absurd that replaces singular values with random noise.
  arXiv  Detail & Related papers  (2026-05-11T19:42:48Z)
• Diff3R: Feed-forward 3D Gaussian Splatting with Uncertainty-aware Differentiable Optimization [76.38917994186733]
  Diff3R is a novel framework that bridges feed-forward prediction and test-time optimization.<n>We show that it can be seamlessly integrated into existing feed-forward 3DGS architectures for both pose-given and pose-free methods.
  arXiv  Detail & Related papers  (2026-04-01T15:40:20Z)
• Two-Stage Optimizer-Aware Online Data Selection for Large Language Models [49.576993784867035]
  We propose a principled-aware framework for gradient-based online data selection and reweighting in large language model fine-tuning.<n>Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the state.<n> Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.
  arXiv  Detail & Related papers  (2026-03-08T21:46:16Z)
• Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods [50.070182958880146]
  We propose a unified framework generalizing descent, quasi-Newton methods, and adaptive methods through the novel notion of preconditioned matrix norms.<n>Within this framework, we provide the first systematic treatment of affine and scale invariance in the matrix- parameterized setting.<n>We introduce two new methods, $ttMuAdam$ and $texttMuAdam-SANIA$, which combine the spectral geometry of Muon with Adam-style preconditioning.
  arXiv  Detail & Related papers  (2025-10-12T19:39:41Z)
• Training Deep Learning Models with Norm-Constrained LMOs [56.00317694850397]
  We propose a new family of algorithms that uses the linear minimization oracle (LMO) to adapt to the geometry of the problem.<n>We demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam.
  arXiv  Detail & Related papers  (2025-02-11T13:10:34Z)
• ROMO: Retrieval-enhanced Offline Model-based Optimization [14.277672372460785]
  Data-driven black-box model-based optimization (MBO) problems arise in a number of practical application scenarios. We propose retrieval-enhanced offline model-based optimization (ROMO) ROMO is simple to implement and outperforms state-of-the-art approaches in the CoMBO setting.
  arXiv  Detail & Related papers  (2023-10-11T15:04:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.