How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization

• URL: http://arxiv.org/abs/2605.14200v1
• Date: Wed, 13 May 2026 23:32:00 GMT
• Title: How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
• Abstract summary: We take a principled step toward resolving this gap by analyzing three different scaling regimes.<n>For each regime, we develop a novel Dynamical Mean Field Theory (DMFT) description of the limiting training dynamics of MoEs.<n>We show that the resulting $$P prescription does not reliably induce monotonic improvement with scale or robust learning-rate transfer.
• Score: 45.69980208532521
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent frontier large language models predominantly rely on Mixture-of-Experts (MoE) architectures. Despite empirical progress, there is still no principled understanding of how hyperparameters should scale with network width $N$, expert width $N_e$, number of experts $M$, sparsity $K$, and depth $L$ to ensure both stability and optimal performance at scale. We take a principled step toward resolving this gap by analyzing three different scaling regimes: (I) co-scaling $N\asymp N_e$, (II) co-scaling $N\asymp M\asymp K$, and (III) full proportional scaling of $N, N_e, M$, and $K$. For each regime, we develop a novel Dynamical Mean Field Theory (DMFT) description of the limiting training dynamics of MoEs that provides a formal foundation for our analysis. Within this framework, we derive the unique parameterization for SGD and Adam satisfying all maximal-update ($μ$) desiderata. We then show that the resulting $μ$P prescription does not reliably induce monotonic improvement with scale or robust learning-rate transfer. We trace these pathologies to scale-dependent observables in the aggregation dynamics, which motivates a refined set of desiderata that we term maximal scale stability. Guided by this principle, we derive a Maximally Scale-Stable Parameterization (MSSP) for both SGD and Adam in all three scaling regimes, and characterize the corresponding limiting dynamics - qualitatively distinct from the $μ$P limit - through a separate DMFT analysis. Experiments verify that MSSP robustly recovers learning rate transfer and monotonic improvement with scale across regimes. Combined with existing depth-scaling theory, these results provide a complete scaling prescription for MoE architectures as a function of width, depth, expert width, and number of experts.

Related papers

• The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning [33.59654714700262]
  We consider the Multiscale Single-Index Model (MSIM) as a stylized model for hierarchical learning with emphscale separation.<n>We perform a fine-grained analysis of the chaos using Wiener Edgeworth expansions.<n>We prove that online SGD on the correlation objective, where all layers evolve in the same timescale, $1 - o_d(1)$ recovery with $n = widetildeO( dK-1)$ samples, recovering the same sample complexity as in the linear counterpart.
  arXiv  Detail & Related papers  (2026-07-03T14:02:42Z)
• Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime [33.80781505782195]
  We present an effective approximate approach to predict the performance of multi-layer perceptrons (MLPs) of fixed depth $L$ on arbitrary high-dimensional data.<n>Even strong representation learning in the proportional limit is encoded in at most $L$ order parameters, determined self-consistently.
  arXiv  Detail & Related papers  (2026-05-28T09:49:54Z)
• Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting [78.96382179408054]
  We introduce bfLeGS, a framework that reformulates density control as a parameterized policy network optimized via Reinforcement Learning (RL)<n>To maintain computational tractability, we derive a closed-form solution that reduces the complexity of reward calculation from $O(N2)$ to $O(N)$.<n>Experiments on the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that bfLeGS significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-01T05:04:49Z)
• Rethinking Language Model Scaling under Transferable Hypersphere Optimization [67.38433364607897]
  We introduce HyperP, the first framework for transferring optimal learning rates across model width, depth, training tokens, and expert-of-Experts (MoE)<n>A single base rate tuned data transfers across all compute budgets under HyperP, yielding $1.58times over a strong Muon baseline at $6times1021$ FLOPs.<n>We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities.
  arXiv  Detail & Related papers  (2026-03-30T17:51:47Z)
• Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization [19.86300998075755]
  We propose a reusable framework for holistic MoE architectural optimization.<n>We first show that FLOPs per token alone is an inadequate fairness metric for MoE models.<n>We then reduce the 16-dimensional architectural search space to two sequential low-dimensional phases.<n>Our framework yields robust scaling laws that map any compute budget to a complete, optimal MoE architecture.
  arXiv  Detail & Related papers  (2026-03-23T11:51:35Z)
• Spectral Condition for $μ$P under Width-Depth Scaling [33.05604496779641]
  We develop a simple and unified spectral framework for $$P under joint width-depth scaling.<n>We show that the proposed spectral $$P preserves condition stable feature learning and enables robust HP transfer under width-depth scaling.
  arXiv  Detail & Related papers  (2026-02-28T08:38:50Z)
• Controlled LLM Training on Spectral Sphere [76.60985966206746]
  We introduce the textbfSpectral Sphere algorithm (SSO), which enforces strict module-wise spectral constraints on both weights and their updates.<n>We observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations.
  arXiv  Detail & Related papers  (2026-01-13T09:59:47Z)
• How to Set the Learning Rate for Large-Scale Pre-training? [73.03133634525635]
  We formalize this investigation into two distinct research paradigms: Fitting and Transfer.<n>Within the Fitting Paradigm, we introduce a Scaling Law for search factor, effectively reducing the search complexity from O(n3) to O(n*C_D*C_) via predictive modeling.<n>We extend the principles of $$Transfer to the Mixture of Experts (MoE) architecture, broadening its applicability to encompass model depth, weight decay, and token horizons.
  arXiv  Detail & Related papers  (2026-01-08T15:55:13Z)
• Quantifying Weighted Morphological Content of Large-Scale Structures via Simulation-Based Inference [0.0]
  We compare the constraining power of two higher-order summary statistics of the large-scale structure (LSS)<n>We find that the Conditional Moments of Derivative (CMD) yields tighter forecasts for $(Omega_m,,sigma_8)$ than the zeroth- to third-order MFs components.
  arXiv  Detail & Related papers  (2025-11-05T16:54:17Z)
• Towards a Comprehensive Scaling Law of Mixture-of-Experts [54.117786590884776]
  We propose a comprehensive and precise joint MoE scaling law that considers all essential factors.<n>Our results demonstrate that the optimal settings for $G$ and $S$ are independent of both the model architecture and data size.<n>Our proposed MoE scaling law could function as an accurate and insightful guidance to facilitate future MoE model design and training.
  arXiv  Detail & Related papers  (2025-09-28T06:35:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.