Muon-C: Operator-Aligned Muon for Convolutional Kernels

• URL: http://arxiv.org/abs/2609.09676v1
• Date: Wed, 09 Sep 2026 03:45:44 GMT
• Title: Muon-C: Operator-Aligned Muon for Convolutional Kernels
• Abstract summary: We introduce Muon, an operator-aligned that represents kernel momentum as frequency-wise channel-transfer matrices.<n>We show that the new geometry arises from combining the block partition and Fourier coordinates.<n>Muon-C reaches 9.87 FID at 40k iterations, compared with 22.26 for unfolded Muon and 51.31 for Adam.
• Score: 5.886986014593717
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Muon replaces matrix momentum with an approximately orthogonal polar direction, but its geometry depends on the matrix representation. For convolution, standard unfolding describes a local patch map rather than the convolution operator. We introduce Muon-C, an operator-aligned optimizer that represents kernel momentum as frequency-wise channel-transfer matrices, polarizes these blocks independently, and uses a critical Fourier grid to return updates exactly to the original finite kernel support. We show that the new geometry arises from combining the block partition and Fourier coordinates. The exact-polar direction is a linear minimization oracle under the critically sampled convolution norm. Its worst-case guarantee relative to the continuous convolution-operator norm is never weaker than unfolding and is strictly stronger for $3\times3$ kernels. On CIFAR-10 flow matching with matched applied-update RMS, Muon-C reaches 9.87 FID at 40k iterations, compared with 22.26 for unfolded Muon and 51.31 for Adam. It reaches their final quality using $0.62\times$ and $0.64\times$ their model FLOPs, respectively. Under equal tuning budgets, Muon-C achieves 3.42 FID. Gains persist across data scales and transfer to classification across convolutional architectures.

Related papers

• Beyond the Matrix Sign: Quadratic Spectral Descent [6.589542614722917]
  We show that curvature can change both the singular values and the singular directions of the optimal update.<n>Experiments on GPT pre-training show that QSD consistently improves validation loss over Muon and recent Muon variants.
  arXiv  Detail & Related papers  (2026-09-07T15:07:24Z)
• Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform [65.89510337046077]
  Equivariant networks embed geometric symmetries as structural priors through weight sharing.<n>Existing implementations unroll the structured weights into dense matrices and dispatch them to generic dense kernels.<n>This is the first time equivariant networks strictly dominate their non-equivariant counterparts along all three axes simultaneously.
  arXiv  Detail & Related papers  (2026-07-23T12:45:21Z)
• Structural Kolmogorov-Arnold Convolutions: Learnable Function on the Values or the Filter Shape as Parameter-Efficient Alternative to Per-Edge Convolutional KANs [76.06235645266621]
  Convolutional Kolmogorov--Arnold Networks (KANs) replace the fixed weights of a convolutional kernel with learnable univariate functions.<n>We argue that the learnable functions are better placed in the emphstructure of the convolution than on each edge.
  arXiv  Detail & Related papers  (2026-06-23T10:00:26Z)
• Muon is Not That Special: Random or Inverted Spectra Work Just as Well [50.969177887027115]
  We demonstrate that precise geometric structure is not key factor affecting performance.<n>We introduce Freon, a family of geometrics based on Schatten (quasi-size)norms.<n>We also introduce Kaon, an absurd that replaces singular values with random noise.
  arXiv  Detail & Related papers  (2026-05-11T19:42:48Z)
• PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation [0.2538209532048867]
  PolarAdamW is a controlled hybrid that preserves Muon's polar spectral-norm control but breaks the gauge-equivariance.<n>On DeiT-Tiny trained from scratch on four independently sampled 100-class subsets of ImageNet-1k, where multiplicity-basis freedom is trivial, PolarAdamW outperforms Muon by +1.93 pp.<n>On SO(3)-equivariant 3D point-cloud regression, where multiplicity-basis freedom is non-trivial, Muon outperforms PolarAdamW at every audited capacity.
  arXiv  Detail & Related papers  (2026-05-08T00:19:13Z)
• SignMuon: Communication-Efficient Distributed Muon Optimization [1.009732307721003]
  We propose Sign-Muon, a 1-bit, matrix-aware iteration that combines majority-vote sign aggregation fromSGD with the polar-based framework of Muon.<n>Across 330AR-10/ResNet-50 configurations Sign-Muon attains the best smoothness (92.15% 4-GPU majority-vote variant) training time at 37% less than other sign-based baselines.
  arXiv  Detail & Related papers  (2026-05-04T13:29:55Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training [0.0]
  We introduce MUD (MomentUm Decorrelation), a complementary whitening approach that replaces Muon's polar update with a triangular (Cholesky-like) whitening surrogate.<n>In terms of time-to-perplexity, MUD yields consistent 10-50% wall-clock improvements over tuned AdamW and Muon.<n>We also demonstrate training a ESM-2 150M protein language model, where MUD matches Muon-level validation perplexity in significantly less wall-clock time.
  arXiv  Detail & Related papers  (2026-03-18T17:37:31Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• Error Feedback for Muon and Friends [80.90330715662961]
  We introduce EF21-Muon, the first communication-efficient, non-Euclidean LMO-based with rigorous convergence guarantees.<n>Our theory covers non-Euclidean smooth and the more general $(L0, L1)$-smooth setting, matching best-known Euclidean rates and enabling faster convergence under suitable norm choices.
  arXiv  Detail & Related papers  (2025-10-01T08:20:08Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.