Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training

• URL: http://arxiv.org/abs/2604.04230v1
• Date: Sun, 05 Apr 2026 19:11:22 GMT
• Title: Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training
• Abstract summary: We model token routing as a congestion game with a single effective parameter, the congestion coefficient gamma_eff, that quantifies the balance-quality tradeoff.<n> Tracking gamma_eff across training checkpoints of two open-source MoE models, we show that early MoE training prioritizes balance while late training prioritizes quality.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We model Mixture-of-Experts (MoE) token routing as a congestion game with a single effective parameter, the congestion coefficient gamma_eff, that quantifies the balance-quality tradeoff. Tracking gamma_eff across training checkpoints of two open-source MoE models, OLMoE-1B-7B (20 checkpoints, with dense sampling in the surge region) and OpenMoE-8B (6 checkpoints), reveals a three-phase trajectory: a surge phase where the router learns to balance load (gamma_eff: 14 to 36-39, peaking in the step 30K-40K region), a stabilization phase where experts specialize under steady balance (B_0: 2.4 to 2.3, steps 100K-400K), and a relaxation phase where the router trades balance for quality as experts differentiate (gamma_eff: 27 to 9, steps 400K-1.2M). This non-monotone trajectory, invisible to post-hoc analysis of converged models, reveals that early MoE training prioritizes balance while late training prioritizes quality. The theoretical framework is honest about its limits: the single-type equilibrium reduces to temperature-scaled softmax (held-out L1: MFG = 0.199 vs. softmax = 0.200). The game is not a better predictor; it reveals what the temperature means and, critically, how that temperature evolves. We complement the dynamics with an effective congestion decomposition, a multi-type extension that improves load prediction via token clustering on all 16 layers (mean: 30%), scope diagnostics (K/M, epsilon_l), and robustness verification across four independent quality estimators (r >= 0.89). All confidence intervals are from bootstrap resampling over 50 independent text batches.

Related papers

• Stochastically Perturbed Weights: Ensembles from Deterministic Machine-Learning Weather Models [0.0]
  Many machine-learning weather models (MLWMs) are deterministic, producing a single forecast with no estimate of its own uncertainty.<n>A growing family of trained-probabilistic models generate calibrated ensembles directly, at the price of a dedicated training run.<n>We ask how much uncertainty can be extracted from a deterministic checkpoint that already exists, without retraining it.
  arXiv  Detail & Related papers  (2026-09-08T08:18:40Z)
• Solution for UCF UrbanTwin LUMPI Track: Sim-to-Real Urban LiDAR 3D Object Detection [54.03436243843709]
  We present our solution to the LUMPI track of the UCF UrbanTwin Sim2Real LiDAR Challenge at the 6th DriveX Workshop, ECCV 2026.<n>The detector must be trained only on synthetic data and is evaluated on 50 held-out real LiDAR frames.<n>A separate 50-frame synthetic submission is evaluated for point-cloud realism.
  arXiv  Detail & Related papers  (2026-09-07T15:03:36Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Multi-Domain Physics-Based MDO of Multirotor UAVs: A Deterministic Framework for Discrete COTS Sizing [1.6114012813668932]
  This paper presents AeroEval, a physics-based, Multidisciplinary Design Optimization (MDO) engine.<n>Engine maps continuous sizing optima to physically purchasable, commercial off-the-shelf (COTS) components.<n>Battery mass is predicted within 7.9% on single-pack platforms.
  arXiv  Detail & Related papers  (2026-07-24T03:52:36Z)
• FlowS: One-Step Motion Prediction via Local Transport Conditioning [3.883772843505009]
  Local transport conditioning makes one-step generative motion prediction practical for safety-critical autonomy.<n>FlowS achieves state-of-the-art Soft mAP (0.4804) and mAP (0.4703) with ensemble at 75, calibrated with single-step inference.
  arXiv  Detail & Related papers  (2026-04-28T19:06:06Z)
• K-GMRF: Kinetic Gauss-Markov Random Field for First-Principles Covariance Tracking on Lie Groups [8.489406212619164]
  We propose K-GMRF, an online, training-free framework for covariance tracking.<n>Our method interprets observations as torques driving a latent angular velocity, propagated via a structure-preserving symplectic integrator.<n>We theoretically prove that this second-order dynamics achieves zero steady-state error under constant rotation.
  arXiv  Detail & Related papers  (2026-03-20T03:16:36Z)
• Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers [5.534626267734823]
  We introduce Variational Mixture-of-Experts Routing (VMoER), a structured Bayesian approach for modelling uncertainty in MoE layers.<n>VMoER confines inference to the expert-selection stage which is typically done by a deterministic routing network.<n>Across tested foundation models, VMoER improves routing under noise by 38%, and increases out-of-distribution AUROC by 12%.
  arXiv  Detail & Related papers  (2026-03-10T10:07:53Z)
• Leap+Verify: Regime-Adaptive Speculative Weight Prediction for Accelerating Neural Network Training [0.0]
  We introduce Leap+Verify to accelerate neural network training.<n>It applies speculative execution -- predicting future model weights and validating predictions before acceptance.<n>We evaluate Leap+Verify on GPT-2 124M and Qwen 2.5-1.5B trained on WikiText-103 across five random seeds.
  arXiv  Detail & Related papers  (2026-02-23T08:01:44Z)
• Deep Time-series Forecasting Needs Kernelized Moment Balancing [56.619037429652984]
  Deep time-series forecasting can be formulated as a distribution balancing problem aimed at aligning the distribution of the forecasts and ground truths.<n>We propose direct forecasting with kernelized moment balancing (KMB-DF)<n>Experiments across multiple models and datasets show that KMB-DF consistently improves forecasting accuracy and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-01-31T13:20:18Z)
• FALCON: Few-step Accurate Likelihoods for Continuous Flows [78.37361800856583]
  We propose Few-step Accurate Likelihoods for Continuous Flows (FALCON), which allows for few-step sampling with a likelihood accurate enough for importance sampling applications.<n>We show FALCON outperforms state-of-the-art normalizing flow models for molecular Boltzmann sampling and is two orders of magnitude faster than the equivalently performing CNF model.
  arXiv  Detail & Related papers  (2025-12-10T18:47:25Z)
• Lane-Frame Quantum Multimodal Driving Forecasts for the Trajectory of Autonomous Vehicles [6.137648436139678]
  Tray forecasting for autonomous driving must deliver accurate, multi-modal calibrated forecasts under tight compute and latency constraints.<n>We propose a compact hybrid quantum architecture that aligns quantum inductive bias with road-scene structure by operating in an ego-centric, lane-aligned frame.<n>In the Open Motion dataset, the model achieves minADE of SI1.94m and minFDE of SI3.56m, consistently outperforming a kinematic baseline with reduced miss rates and strong recall.
  arXiv  Detail & Related papers  (2025-11-21T07:00:07Z)
• MeanFlow Transformers with Representation Autoencoders [71.45823902973349]
  MeanFlow (MF) is a diffusion-motivated generative model that enables efficient few-step generation by learning long jumps directly from noise to data.<n>We develop an efficient training and sampling scheme for MF in the latent space of a Representation Autoencoder (RAE)<n>We achieve a 1-step FID of 2.03, outperforming vanilla MF's 3.43, while reducing sampling GFLOPS by 38% and total training cost by 83% on ImageNet 256.
  arXiv  Detail & Related papers  (2025-11-17T06:17:08Z)
• Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model [100.86587937568832]
  Ring-1T is the first open-source, state-of-the-art thinking model with a trillion-scale parameter.<n>It features 1 trillion total parameters and activates approximately 50 billion per token.
  arXiv  Detail & Related papers  (2025-10-21T17:46:14Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.