Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

• URL: http://arxiv.org/abs/2606.16899v1
• Date: Mon, 15 Jun 2026 16:09:02 GMT
• Title: Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
• Abstract summary: Matrix baseds such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink when using standard constant weight decay.<n>We propose Hyperball, a simple wrapper that addresses this issue.<n>On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20-30% token equivalent speedup over weight decay baselines.
• Score: 67.29294986811314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Matrix based optimizers such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose Hyperball, a simple optimizer wrapper that addresses this issue. Given a base optimizer such as Adam or Muon, Hyperball sets the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20--30% token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay. This method is motivated by prior theory showing that training with weight decay leads to an equilibrium weight norm that only depends on the training hyperparameters. Through this mechanism, the weight decay then decides the angular learning rate, i.e. how fast the direction of the weight matrix changes.

Related papers

• Scale Weight Decay and Train Better [0.0]
  We propose to scale weight decay by a fraction of the peak learning rate $/_max$.<n>We prove that this preserves the stationarity guarantees of the corresponding unregularized methods for both a gradient descent and a non-Euclidean Muon.<n>Using a steady-state analysis, we explain why under standard weight decay the weight norm shrinks steadily as training proceeds, whereas under scaled weight decay it settles to roughly constant value.
  arXiv  Detail & Related papers  (2026-07-26T17:55:40Z)
• Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors [110.10775872372047]
  We propose emphMagnitude-Direction-Direction (MD) Decoupling.<n>It factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates.<n>Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts models.
  arXiv  Detail & Related papers  (2026-06-24T15:40:26Z)
• GQA-μP: The maximal parameterization update for grouped query attention [64.99972873396649]
  We show how to derivation P scalings for grouped-query attention (GQA)<n>We also demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter.
  arXiv  Detail & Related papers  (2026-05-14T18:03:16Z)
• Rethinking Language Model Scaling under Transferable Hypersphere Optimization [67.38433364607897]
  We introduce HyperP, the first framework for transferring optimal learning rates across model width, depth, training tokens, and expert-of-Experts (MoE)<n>A single base rate tuned data transfers across all compute budgets under HyperP, yielding $1.58times over a strong Muon baseline at $6times1021$ FLOPs.<n>We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities.
  arXiv  Detail & Related papers  (2026-03-30T17:51:47Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales [55.91454326946738]
  We study how the optimal learning rate and weight decay should scale with model width and depth for a wide range of languages.<n>We find that scaling the learning rate according to $$P improves transfer, but can still suffer from significant finite-width deviations.<n>For compute-optimal scaling, we find scaling independent weight decay as $1/mathrmwidth$ is nearly optimal across languages.
  arXiv  Detail & Related papers  (2025-12-05T11:03:41Z)
• Dion: Distributed Orthonormalized Updates [27.66769374729482]
  We introduce Dion (Distributed Orthonormalization), a scalable and efficient update rule.<n>It replaces Newton-Schulz iteration with amortized power iteration on a momentum buffer.<n>The rank-fraction parameter with error feedback enables low-rank updates that balance quality with significant cost savings.
  arXiv  Detail & Related papers  (2025-04-07T17:49:37Z)
• How to set AdamW's weight decay as you scale model and dataset size [29.980824873382833]
  We show that weights learned by AdamW can be understood as an exponential moving average (EMA) of recent updates.<n>This gives critical insights for how to set the weight decay in AdamW, and how the weight decay should scale with model and dataset size.
  arXiv  Detail & Related papers  (2024-05-22T14:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.