PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation

• URL: http://arxiv.org/abs/2605.07067v1
• Date: Fri, 08 May 2026 00:19:13 GMT
• Title: PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation
• Abstract summary: PolarAdamW is a controlled hybrid that preserves Muon's polar spectral-norm control but breaks the gauge-equivariance.<n>On DeiT-Tiny trained from scratch on four independently sampled 100-class subsets of ImageNet-1k, where multiplicity-basis freedom is trivial, PolarAdamW outperforms Muon by +1.93 pp.<n>On SO(3)-equivariant 3D point-cloud regression, where multiplicity-basis freedom is non-trivial, Muon outperforms PolarAdamW at every audited capacity.
• Score: 0.2538209532048867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Muon's matrix-level update couples two distinct effects: spectral control via a polar map, and equivariance under orthogonal changes of multiplicity-space basis (Schur gauge-equivariance). We separate them with PolarAdamW, a controlled hybrid that preserves Muon's polar spectral-norm control but breaks the gauge-equivariance, since AdamW's coordinatewise preconditioner is basis-dependent. Algorithmically, PolarAdamW applies Muon's Newton-Schulz polar map to AdamW's preconditioned direction rather than to raw momentum, at per-iteration wall-time comparable to Muon. We prove that Muon's polar step is Schur gauge-equivariant on multiplicity matrices while AdamW's coordinatewise step is not. On DeiT-Tiny trained from scratch on four independently sampled 100-class subsets of ImageNet-1k, where multiplicity-basis freedom is trivial, PolarAdamW outperforms Muon by +1.93 pp in test accuracy on average and AdamW by +9.5 pp; under the 300-epoch DeiT-style recipe, it remains ahead of Muon by +1.37 pp and AdamW by +5.80 pp on average. On SO(3)-equivariant 3D point-cloud regression, where multiplicity-basis freedom is non-trivial, the ordering reverses: Muon outperforms PolarAdamW at every audited capacity, and the gap widens with capacity. Both matrix-polar optimisers continue to outperform AdamW. This double dissociation separates spectral control from Schur gauge-equivariance: the first composes well with AdamW preconditioning on standard transformers, while the second becomes consequential when multiplicity-basis freedom is structurally non-trivial.

Related papers

• Muon-C: Operator-Aligned Muon for Convolutional Kernels [5.886986014593717]
  We introduce Muon, an operator-aligned that represents kernel momentum as frequency-wise channel-transfer matrices.<n>We show that the new geometry arises from combining the block partition and Fourier coordinates.<n>Muon-C reaches 9.87 FID at 40k iterations, compared with 22.26 for unfolded Muon and 51.31 for Adam.
  arXiv  Detail & Related papers  (2026-09-09T03:45:44Z)
• Equivariance Breaks the Learning Rate [24.409404298953234]
  Equivariant networks are commonly trained with Adam.<n>We identify one source of this difference inside equivariant linear layers.<n>We show that blockwise step control and momentum accumulation account for much of Muon's advantage.
  arXiv  Detail & Related papers  (2026-09-08T07:52:03Z)
• An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies [3.9617623336464636]
  Muon and related matrix-signs are increasingly used to pre-train large language models.<n>This report proposes a unified framework built on a single idealizing assumption.<n>We study three systems trained with Muon: a nanoGPT feed-forward projection, a 64-expert mixture-of-experts router, and the query/key projections of a bf16 FlashAttention block.
  arXiv  Detail & Related papers  (2026-07-22T05:39:00Z)
• Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors [110.10775872372047]
  We propose emphMagnitude-Direction-Direction (MD) Decoupling.<n>It factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates.<n>Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts models.
  arXiv  Detail & Related papers  (2026-06-24T15:40:26Z)
• Why Muon Outperforms Adam: A Curvature Perspective [49.85900116602357]
  Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
  arXiv  Detail & Related papers  (2026-06-03T09:40:30Z)
• How Much Orthogonalization Does Muon Need? [20.05604748384174]
  Muons improve neural-network training by replacing ill-conditioned momentum updates with approximately semi-orthogonal updates.<n>We use a relaxed cubic Newton--Schulz schedule derived directly for Muon's low precision singular value band.<n>We find that training quality is not governed monotonically by polar-decomposition accuracy.
  arXiv  Detail & Related papers  (2026-05-29T21:26:25Z)
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning [262.0414794728708]
  We describe the pre-training, post-training, and quantization of Nemotron 3 Super.<n>Nemotron 3 Super is a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model.
  arXiv  Detail & Related papers  (2026-04-14T07:02:32Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training [0.0]
  We introduce MUD (MomentUm Decorrelation), a complementary whitening approach that replaces Muon's polar update with a triangular (Cholesky-like) whitening surrogate.<n>In terms of time-to-perplexity, MUD yields consistent 10-50% wall-clock improvements over tuned AdamW and Muon.<n>We also demonstrate training a ESM-2 150M protein language model, where MUD matches Muon-level validation perplexity in significantly less wall-clock time.
  arXiv  Detail & Related papers  (2026-03-18T17:37:31Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• MuonBP: Faster Muon via Block-Periodic Orthogonalization [24.232069944820513]
  We show how to adjust the learning rate from the baseline to MuonBP and give guarantees for this algorithm.<n>When training an 8B model with eight-way tensor tensor and ZeRO statewiseing, MuonBP achieves 8% Muon with no degradation in performance.
  arXiv  Detail & Related papers  (2025-10-19T19:56:05Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Error Feedback for Muon and Friends [80.90330715662961]
  We introduce EF21-Muon, the first communication-efficient, non-Euclidean LMO-based with rigorous convergence guarantees.<n>Our theory covers non-Euclidean smooth and the more general $(L0, L1)$-smooth setting, matching best-known Euclidean rates and enabling faster convergence under suitable norm choices.
  arXiv  Detail & Related papers  (2025-10-01T08:20:08Z)
• Kourkoutas-Beta: A Sunspike-Driven Adam Optimizer with Desert Flair [0.0]
  Transformer neural networks are increasingly used for physics-based problems.<n>In data-driven PDE surrogates, training samples from varying boundary and initial conditions can cause erratic losses and spiky gradients.<n>We introduce Kourkoutas-Beta, an Adam-style discount where the fixed second-moment beta2 is replaced by a layer-wise dynamic value.
  arXiv  Detail & Related papers  (2025-08-18T15:16:54Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.