A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router

• URL: http://arxiv.org/abs/2605.29121v1
• Date: Wed, 27 May 2026 21:29:30 GMT
• Title: A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router
• Abstract summary: We propose a minimal dynamical model of adaptive softmax routing for a two-expert Mixture-of-Experts layer.<n>We derive exact parametric equations for the bifurcation set and the local normal form of the cusp catastrophe.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a minimal dynamical model of adaptive softmax routing for a two-expert Mixture-of-Experts (MoE) layer. The model is obtained as a mean-field limit of a discrete reinforcement rule: the selected expert receives a small score increment, while all scores undergo regularizing decay. In the symmetric case the limiting system has a supercritical pitchfork bifurcation: for weak feedback there is a unique stable balanced state, whereas above a critical feedback strength two stable asymmetric states appear. When an external asymmetry is added, the pitchfork unfolds into a pair of fold bifurcations forming a cusp in the control-parameter plane. We derive exact parametric equations for the bifurcation set and the local normal form of the cusp catastrophe. Numerical experiments connect this picture to empirical expert load, a small trainable MoE model, hard top-1 PyTorch routing, and a small classification experiment on digits. The results provide a controlled low-dimensional mechanism for abrupt transitions to load imbalance in adaptive MoE routers.

Related papers

• Scalable Lindblad Noise Learning via Stochastic Tensor-Network Simulation [48.887627688666974]
  Learning dissipation rates in large-scale open quantum systems is a major obstacle for near-term quantum technologies.<n>Here, we propose a scalable noise-learning framework for Lindblad dissipation rates that combines a simulation method, the Jump Method (TJM)<n>We demonstrate the approach on two noise models in the Ising model, in which independent dissipation rates are learned for each site up to $N_mathrmsite=16$.<n>This combination of scalable numerics and rigorous theoretical guarantees positions TJM-based noise learning as a practical foundation for characterizing dissipation in large quantum devices.
  arXiv  Detail & Related papers  (2026-08-25T15:07:11Z)
• Low-Depth and Noise-Resilient Quantum State Preparation for Partial Differential Equations via Virtual Rz [0.0]
  Exact state-preparation decompositions can reach unit fidelity ideally, but their two-qubit depth grows quickly and causes severe fidelity loss on NISQ hardware.<n>We propose a low-depth, hardware-aware variational ansatz tailored to smooth, weakly entangled, near-real target distributions typical of damped PDE dynamics.<n>Results indicate that virtual-Rz-enabled, low-depth circuits provide a practical, noise-resilient state-preparation primitive for PDE solvers on NISQ and early FTQC hardware.
  arXiv  Detail & Related papers  (2026-08-18T01:11:10Z)
• Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training [56.323119575322146]
  Extra-Merge consistently outperforms standard merging baselines in experiments across GPT-2 and LLaMA families.<n>It yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon citepjordan2024muon.
  arXiv  Detail & Related papers  (2026-05-26T02:48:34Z)
• $φ$-Balancing for Mixture-of-Experts Training [20.061328126956028]
  Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability.<n>We propose $$-balancing, a principled framework that directly targets population-level expert balance by minimizing a strictly convex, symmetric, and differentiable potential of the expected routing distribution.
  arXiv  Detail & Related papers  (2026-05-14T20:39:28Z)
• Efficient Adjoint Matching for Fine-tuning Diffusion Models [47.05085426361986]
  We propose EAM, which substantially improves training efficiency by reformulating the SOC problem with a textit base drift and a textitterminal cost.<n>EAM converges up to 4x faster than AM and matches or surpasses it across various metrics including PickScore, ImageReward, HPSv2.1, CLIPScore and Aesthetics.
  arXiv  Detail & Related papers  (2026-05-12T03:55:12Z)
• Neural Control: Adjoint Learning Through Equilibrium Constraints [7.41073008855977]
  We propose a boundary-control framework that computes trajectory-dependent, memory-efficient proxy gradients.<n>We show improved performance over gradient-free baselines such as SPSA and CEM.
  arXiv  Detail & Related papers  (2026-05-05T02:19:37Z)
• Boundary Mass and the Soft-to-Hard Limit in Mixture-of-Experts [0.0]
  We study the singularity at the population level for squared-loss MoE regression.<n>The central object is the emphboundary mass, namely the probability that the top two router scores are separated by only a small margin.<n>We show that the zero-temperature limit is controlled by a thin geometric layer around routing interfaces, not by the full input space.
  arXiv  Detail & Related papers  (2026-05-04T01:07:21Z)
• Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts [74.40169987564724]
  Expert parallelism (EP) is designed to scale MoE models by distributing experts across multiple devices.<n>Under extreme imbalance, EP can funnel a disproportionate number of tokens to a small number of experts, leading to compute- and memory-bound failures.<n>We propose Least-Loaded Expert Parallelism (LLEP), a novel EP algorithm that dynamically reroutes excess tokens and associated expert parameters from overloaded devices to underutilized ones.
  arXiv  Detail & Related papers  (2026-01-23T18:19:15Z)
• Lane-Frame Quantum Multimodal Driving Forecasts for the Trajectory of Autonomous Vehicles [6.137648436139678]
  Tray forecasting for autonomous driving must deliver accurate, multi-modal calibrated forecasts under tight compute and latency constraints.<n>We propose a compact hybrid quantum architecture that aligns quantum inductive bias with road-scene structure by operating in an ego-centric, lane-aligned frame.<n>In the Open Motion dataset, the model achieves minADE of SI1.94m and minFDE of SI3.56m, consistently outperforming a kinematic baseline with reduced miss rates and strong recall.
  arXiv  Detail & Related papers  (2025-11-21T07:00:07Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Fast Scrambling in the Hyperbolic Ising Model [45.9982965995401]
  We investigate many-body chaos and scrambling in the Hyperbolic Ising model, a mixed-field Ising model living in the background of AdS2.<n>We show that this model with only local site-dependent nearest neighbor interactions is maximally chaotic, can be classified as a fast scrambler, and saturates the Maldacena-Shenker-Stanford (MSS) bound on chaos for a certain set of parameters.
  arXiv  Detail & Related papers  (2025-02-28T19:00:07Z)
• CWF: Consolidating Weak Features in High-quality Mesh Simplification [50.634070540791555]
  We propose a smooth functional that simultaneously considers all of these requirements. The functional comprises a normal anisotropy term and a Centroidal Voronoi Tessellation (CVT) energy term.
  arXiv  Detail & Related papers  (2024-04-24T05:37:17Z)
• Variational quantum simulation of critical Ising model with symmetry averaging [0.2578242050187029]
  We investigate the use of deep multi-scale entanglement renormalization circuits as a variational ansatz for ground states of gapless systems. We find that DMERA strongly outperforms a standard QAOA-style ansatz, and that a major source of systematic error in correlation functions approximated using DMERA is the breaking of the translational and Kramers-Wannier symmetries of the transverse-field Ising model.
  arXiv  Detail & Related papers  (2022-10-26T21:37:14Z)
• Noise-resilient Edge Modes on a Chain of Superconducting Qubits [103.93329374521808]
  Inherent symmetry of a quantum system may protect its otherwise fragile states. We implement the one-dimensional kicked Ising model which exhibits non-local Majorana edge modes (MEMs) with $mathbbZ$ parity symmetry. MEMs are found to be resilient against certain symmetry-breaking noise owing to a prethermalization mechanism.
  arXiv  Detail & Related papers  (2022-04-24T22:34:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.