Dion3: Full-Stack Orthogonal Updates

• URL: http://arxiv.org/abs/2608.11612v1
• Date: Wed, 12 Aug 2026 03:42:40 GMT
• Title: Dion3: Full-Stack Orthogonal Updates
• Abstract summary: We present Dion3, a revision of Muon that targets this overhead at every level of the stack.<n>Dion3 improves on the loss achieved by Muon but reduces step time by up to 6x.
• Score: 39.76588854070093
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Muon optimizer incurs a significant overhead cost due to its cubic-time Newton-Schulz orthogonalization step. When weights are sharded, communication overhead compounds this computational cost, eroding the benefits of Muon in many settings. We present Dion3, a revision of Muon that targets this overhead at every level of the stack. Our Gram Newton-Schulz algorithm reduces the FLOP cost of orthogonalization, our CuteDSL kernels accelerate it by exploiting symmetry, and our megabatching strategy reduces communication overhead. Moreover, we propose a simple change to the update rule that cuts costs even further: selecting only a fraction of the momentum matrix's rows to orthogonalize at each step. This update rule improves on Dion (another "compressed" version of Muon), in both speed and performance. Overall, Dion3 matches or improves on the loss achieved by Muon but reduces optimizer step time by up to 6x. Dion3 is available via the dion package (https://github.com/microsoft/dion) as a drop-in replacement for Muon.

Related papers

• Muon Meets Mamba: Spectral Optimization for State Space Models [39.146761527401424]
  We compare Muon with AdamW on Mamba-2 130M under a controlled protocol that varies only which weight groups are trained with Muon.<n>Muon on the output projection alone beats Muon on the input projection or on both.<n> Conditioning does not explain the gain. Muon lowers the condition number of whichever projection it trains, but the better-conditioned input projection is not the one that helps.
  arXiv  Detail & Related papers  (2026-08-04T17:10:47Z)
• Why Muon Outperforms Adam: A Curvature Perspective [49.85900116602357]
  Muon improves training efficiency over Adam in large language-model training by about two times.<n>Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective.
  arXiv  Detail & Related papers  (2026-06-03T09:40:30Z)
• LionMuon: Alternating Spectral and Sign Descent for Efficient Training [48.69598464267559]
  We propose LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged cost.<n>It alternates between Lion's and Muon's updates on a fixed period P, sharing a single dual-EMA momentum buffer between them.<n>A single-EMA variant, SignMuon, by itself already outperforms pure Muon.
  arXiv  Detail & Related papers  (2026-05-19T13:07:59Z)
• MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models [45.11415579822849]
  We study generalization error of the Muon based on algorithmic stability and mathematical induction.<n>We propose an effective mixed Muon (MiMuon) by cautiously using gradient, which is a hybrid of Muon and momentum-based SGDs.<n>Our MiMuon algorithm has the same convergence rate of $O(frac1NTbig)$ as the Muon algorithm.
  arXiv  Detail & Related papers  (2026-05-19T09:56:27Z)
• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning [18.570226339282296]
  Muon has emerged as a promising computation for large-scale foundation model pre-training.<n>We propose Muon$2$, a memory-efficient factorized variant that preserves most of the gains of Muon$2$ with negligible memory overhead.
  arXiv  Detail & Related papers  (2026-04-11T00:27:40Z)
• Dion2: A Simple Method to Shrink Matrix in Muon [19.766325230655173]
  We introduce Dion2, a much simpler method for shrinking the matrix involved in Muon's iteration compared to prior approaches.<n>At a high level, Dion2 selects a fraction of rows or columns at each and orthonormalizes only those.
  arXiv  Detail & Related papers  (2025-12-01T16:58:10Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Dion: Distributed Orthonormalized Updates [27.66769374729482]
  We introduce Dion (Distributed Orthonormalization), a scalable and efficient update rule.<n>It replaces Newton-Schulz iteration with amortized power iteration on a momentum buffer.<n>The rank-fraction parameter with error feedback enables low-rank updates that balance quality with significant cost savings.
  arXiv  Detail & Related papers  (2025-04-07T17:49:37Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.