Muon Dynamics as a Spectral Wasserstein Flow

• URL: http://arxiv.org/abs/2604.04891v1
• Date: Mon, 06 Apr 2026 17:41:12 GMT
• Title: Muon Dynamics as a Spectral Wasserstein Flow
• Abstract summary: We introduce a family of Wasserstein norms on positive semidefinite matrices.<n>We prove the equivalence between the static and dynamic Benamou-Brenier formulations.<n>We identify exact gradientity associated with flow models.
• Score: 21.689846521201588
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gradient normalization is central in deep-learning optimization because it stabilizes training and reduces sensitivity to scale. For deep architectures, parameters are naturally grouped into matrices or blocks, so spectral normalizations are often more faithful than coordinatewise Euclidean ones; Muon is the main motivating example of this paper. More broadly, we study a family of spectral normalization rules, ranging from ordinary gradient descent to Muon and intermediate Schatten-type schemes, in a mean-field regime where parameters are modeled by probability measures. We introduce a family of Spectral Wasserstein distances indexed by a norm gamma on positive semidefinite matrices. The trace norm recovers the classical quadratic Wasserstein distance, the operator norm recovers the Muon geometry, and intermediate Schatten norms interpolate between them. We develop the static Kantorovich formulation, prove comparison bounds with W2, derive a max-min representation, and obtain a conditional Brenier theorem. For Gaussian marginals, the problem reduces to a constrained optimization on covariance matrices, extending the Bures formula and yielding a closed form for commuting covariances in the Schatten family. For monotone norms, including all Schatten cases, we prove the equivalence between the static and dynamic Benamou-Brenier formulations, deduce that the resulting transport cost is a genuine metric equivalent to W2 in fixed dimension, and show that the induced Gaussian covariance cost is also a metric. We then interpret the associated normalized continuity equation as a Spectral Wasserstein gradient flow, identify its exact finite-particle counterpart as a normalized matrix flow, obtain first geodesic-convexity results, and show how positively homogeneous mean-field models induce a spectral unbalanced transport on the sphere.

Related papers

• Quotient Dynamics, Effective Curvature, and Implicit Bias in Positive Quadratic Networks [0.0]
  Positive quadratic networks admit the lowrank representation f_U(x)=xtop UUtop x, where UinmathbbRdtimes r is identifiable only up to right multiplication.<n>We study how this quotient structure governs training dynamics, curvature, recovery, and bias.
  arXiv  Detail & Related papers  (2026-07-28T12:04:46Z)
• Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation [3.0459410574367314]
  We introduce Mean Root Square Normalization (MRSNorm)<n>MRSNorm mathematically inverts the traditional scaling paradigm.<n>We analytically demonstrate that this constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity.
  arXiv  Detail & Related papers  (2026-07-20T11:10:51Z)
• Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds [14.996920125049002]
  Muon and related norm-constrained matrix oracles have become central to large-scale learning problems.<n>We introduce intrinsic Muon (iMuon), a unified framework that yields closed-form updates on the fixed-rank, SPD, Stiefel, and Grassmann matrices.<n> Experiments on LoRA finetuning of LLMs, image classification, and subspace learning illustrate the proposed approach.
  arXiv  Detail & Related papers  (2026-05-10T00:39:13Z)
• On Higher-Order Geometric Refinements of Classical Covariance Asymptotics: An Approach via Intrinsic and Extrinsic Information Geometry [0.0]
  In curved models, including mixtures, curved exponential families, latent-variable models, and manifold-root parameter spaces, finite-sample behavior can deviate systematically from predictions.<n>We develop a coordinate-invariant, curvature-aware refinement by viewing a regular parametric family as a ((,g)) with Fisher-constrained-Rao metric.<n>We describe the resolved metric, the role of the real log canonical threshold in learning rates and posterior mean-squared error, and a curvature-based covariance expansion on the resolved space that recovers the regular theory as a special case.
  arXiv  Detail & Related papers  (2026-04-14T13:40:13Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds [0.5729426778193398]
  We study the emergence of multi-step reasoning in deep Transformer language models through a geometric and statistical-physics lens.<n>We formalize the forward pass as a discrete coarse-graining map and relate the appearance of stable "concept basins" to fixed points of this renormalization-like dynamics.<n>The resulting low-entropy regime is characterized by a spectral tail collapse and by the formation of transient, reusable object-like structures in representation space.
  arXiv  Detail & Related papers  (2026-01-16T23:11:02Z)
• Solving a Nonlinear Eigenvalue Equation in Quantum Information Theory: A Hybrid Approach to Entanglement Quantification [0.0]
  We present a hybrid analytical and numerical framework for evaluating the geometric measure of entanglement.<n>We make the coupled nonlinear eigenstructure explicit by proving the equal multiplier stationarity.<n>The resulting hybrid solver reproduces the exact optimum for standard three qubit benchmarks.
  arXiv  Detail & Related papers  (2025-11-14T13:36:47Z)
• Spectrum-Aware Debiasing: A Modern Inference Framework with Applications to Principal Components Regression [1.342834401139078]
  We introduce SpectrumAware Debiasing, a novel method for high-dimensional regression. Our approach applies to problems with structured, heavy tails, and low-rank structures. We demonstrate our method through simulated and real data experiments.
  arXiv  Detail & Related papers  (2023-09-14T15:58:30Z)
• The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks [117.93273337740442]
  We show that gradient descent converges to a uniform margin classifier on the training data with an $exp(-Omega(log2 t))$ convergence rate. We also show that batch normalization has an implicit bias towards a patch-wise uniform margin.
  arXiv  Detail & Related papers  (2023-06-20T16:58:00Z)
• Nonconvex Stochastic Scaled-Gradient Descent and Generalized Eigenvector Problems [98.34292831923335]
  Motivated by the problem of online correlation analysis, we propose the emphStochastic Scaled-Gradient Descent (SSD) algorithm. We bring these ideas together in an application to online correlation analysis, deriving for the first time an optimal one-time-scale algorithm with an explicit rate of local convergence to normality.
  arXiv  Detail & Related papers  (2021-12-29T18:46:52Z)
• Spectral clustering under degree heterogeneity: a case for the random walk Laplacian [83.79286663107845]
  This paper shows that graph spectral embedding using the random walk Laplacian produces vector representations which are completely corrected for node degree. In the special case of a degree-corrected block model, the embedding concentrates about K distinct points, representing communities.
  arXiv  Detail & Related papers  (2021-05-03T16:36:27Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Doubly-Stochastic Normalization of the Gaussian Kernel is Robust to Heteroskedastic Noise [3.5429774642987915]
  We show that the doubly-stochastic normalization of the Gaussian kernel with zero main diagonal is robust to heteroskedastic noise. We provide examples of simulated and experimental single-cell RNA sequence data with intrinsic heteroskedasticity.
  arXiv  Detail & Related papers  (2020-05-31T01:31:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.