Rethinking Language Model Scaling under Transferable Hypersphere Optimization

• URL: http://arxiv.org/abs/2603.28743v2
• Date: Sun, 05 Apr 2026 02:15:47 GMT
• Title: Rethinking Language Model Scaling under Transferable Hypersphere Optimization
• Abstract summary: We introduce HyperP, the first framework for transferring optimal learning rates across model width, depth, training tokens, and expert-of-Experts (MoE)<n>A single base rate tuned data transfers across all compute budgets under HyperP, yielding $1.58times over a strong Muon baseline at $6times1021$ FLOPs.<n>We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities.
• Score: 67.38433364607897
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling laws for large language models depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizers, and they do not structurally prevent training instability at scale. Recent hypersphere optimization methods constrain weight matrices to a fixed-norm hypersphere, offering a promising alternative for more stable scaling. We introduce HyperP (Hypersphere Parameterization), the first framework for transferring optimal learning rates across model width, depth, training tokens, and Mixture-of-Experts (MoE) granularity under the Frobenius-sphere constraint with the Muon optimizer. We prove that weight decay is a first-order no-op on the Frobenius sphere, show that Depth-$μ$P remains necessary, and find that the optimal learning rate follows the same data-scaling power law with the "magic exponent" 0.32 previously observed for AdamW. A single base learning rate tuned at the smallest scale transfers across all compute budgets under HyperP, yielding $1.58\times$ compute efficiency over a strong Muon baseline at $6\times10^{21}$ FLOPs. Moreover, HyperP delivers transferable stability: all monitored instability indicators, including $Z$-values, output RMS, and activation outliers, remain bounded and non-increasing under training FLOPs scaling. We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities for improved granularity scaling, and show that hypersphere optimization enables substantially larger auxiliary load-balancing weights, yielding both strong performance and good expert balance. We release our training codebase at https://github.com/microsoft/ArchScale.

Related papers

• Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts [2.2554080765044717]
  Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.<n>We propose a compute-efficient, two-step hyperparameter transfer framework that estimates optimal learning rates for training large MoE models.
  arXiv  Detail & Related papers  (2026-08-20T13:57:43Z)
• Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization [67.29294986811314]
  Matrix baseds such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink when using standard constant weight decay.<n>We propose Hyperball, a simple wrapper that addresses this issue.<n>On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20-30% token equivalent speedup over weight decay baselines.
  arXiv  Detail & Related papers  (2026-06-15T16:09:02Z)
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate [10.599439539657787]
  We show that Maximal Update ($$P) offers high-quality learning rate transfer relative to standard parameterization.<n>We find that the overwhelming benefit of $$P relative to standard parameterization arises simply from maximizing the learning rate of the embedding layer.
  arXiv  Detail & Related papers  (2026-05-20T17:59:40Z)
• Extending $μ$P: Spectral Conditions for Feature Learning Across Optimizers [3.5708391029226885]
  We propose a novel framework to derive $$P for a broader class of derivations, including AdamW, AD, LAMB, Sophia, Shampoo and Muon.<n>We implement our $$Ps on multiple benchmark models and demonstrate zero-shot learning rate transfer across increasing model width.
  arXiv  Detail & Related papers  (2026-02-24T14:17:51Z)
• TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers [24.534939825452884]
  TrasMuon preserves the near-isometric geometry of Muon while stabilizing magnitudes through (i) global RMS calibration and (ii) energy-based trust-region clipping.<n>We demonstrate that while reintroducing adaptive scaling improves optimization efficiency, it typically exacerbates instability due to high-energy outliers.<n> Empirical experiments on vision and language models demonstrate that TrasMuon converges faster than baselines.
  arXiv  Detail & Related papers  (2026-02-13T22:11:59Z)
• Controlled LLM Training on Spectral Sphere [76.60985966206746]
  We introduce the textbfSpectral Sphere algorithm (SSO), which enforces strict module-wise spectral constraints on both weights and their updates.<n>We observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations.
  arXiv  Detail & Related papers  (2026-01-13T09:59:47Z)
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales [55.91454326946738]
  We study how the optimal learning rate and weight decay should scale with model width and depth for a wide range of languages.<n>We find that scaling the learning rate according to $$P improves transfer, but can still suffer from significant finite-width deviations.<n>For compute-optimal scaling, we find scaling independent weight decay as $1/mathrmwidth$ is nearly optimal across languages.
  arXiv  Detail & Related papers  (2025-12-05T11:03:41Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining [59.369484219304866]
  We conduct an unprecedented empirical investigation training over 3,700 Large Language Models (LLMs) from scratch across 100 trillion tokens.<n>We establish a universal Scaling Law for hyperparameter optimization in LLM Pre-training, called Step Law.<n>Our estimated optima deviates from the global best performance found via exhaustive search by merely 0.094% on the test set.
  arXiv  Detail & Related papers  (2025-03-06T18:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.