Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement

• URL: http://arxiv.org/abs/2607.08782v1
• Date: Sat, 13 Jun 2026 04:31:49 GMT
• Title: Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement
• Abstract summary: Director is a new distributed MoE serving system that minimizes end-to-end latency via prediction-driven expert placement.<n>We implement a prototype and demonstrate, through extensive experiments, a reduction in end-to-end latency of $11sim55%$ for popular MoE models.
• Score: 24.858126206720105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models. Its efficiency depends on the communication and computation latencies of the GPUs, which are linked to the placement of experts in the GPUs. Existing works for optimizing expert placement focus on leveraging past requests' expert activation patterns. However, they demonstrate deficiencies facing diverse and rapidly changing request patterns, calling for an online, proactive approach. Implementing such an approach requires addressing several challenges: the uncertainty associated with incoming requests' expert activation, the cost of expert migration, and the NP-hard complexity in optimization. Therefore, we present Director, a new distributed MoE serving system that minimizes end-to-end latency via prediction-driven, online expert placement. Director uses either a lightweight cascaded predictor or a low-bit quantized replica for expert activation patterns of incoming requests. An online migration module then enacts the changes with near-zero downtime by executing migrations in compute-bound phases, keeping disruption bounded. At its core, a relaxation-based expert placement optimizer operates under capacity constraints, runs in polynomial time, and achieves a $(1+ε)$ approximation ratio. Finally, we implement a prototype and demonstrate, through extensive experiments, a reduction in end-to-end latency of $11\sim55\%$ for popular MoE models (e.g., Mistral, DeepSeek and Qwen) compared to existing work.

Related papers

• A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference [10.085075886001665]
  Mixture-of-Experts (MoE) based large language models (LLMs) have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost.<n>We propose ST-MoE, atemporal expert prefetching framework that proactively stages experts ahead of use to overlap expert loading with ongoing computation.
  arXiv  Detail & Related papers  (2026-06-13T20:09:54Z)
• MoEless: Efficient MoE LLM Serving via Serverless Computing [14.858661450986794]
  MoEless is the first serverless MoE serving framework that mitigates expert load imbalance and accelerates inference via serverless experts.<n>MoEless employs lightweight, layer-aware predictors to accurately estimate incoming expert load distributions and proactively identify stragglers.<n>MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions.
  arXiv  Detail & Related papers  (2026-03-06T14:58:16Z)
• Climber-Pilot: A Non-Myopic Generative Recommendation Model Towards Better Instruction-Following [19.550149895505683]
  We present Climber-Pilot, a unified generative retrieval framework.<n>We introduce Time-Aware Multi-Item Prediction (TAMIP), a novel training paradigm designed to mitigate inherent myopia in generative retrieval.<n>We also propose Condition-Guided Sparse Attention (CGSA), which incorporates business constraints directly into the generative process via sparse attention.
  arXiv  Detail & Related papers  (2026-02-14T03:46:06Z)
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning [83.66308307152808]
  We propose StAbilized Mixture-of-Experts (SAME) for Multimodal Continual Instruction Tuning (MCIT)<n>SAME stabilizes expert selection by decomposing routing dynamics into subspaces and updating only task-relevant directions.<n>It also introduces adaptive expert activation to freeze selected experts during training, reducing redundant and cross-task interference.
  arXiv  Detail & Related papers  (2026-02-02T11:47:06Z)
• Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inference [49.141930185079325]
  We propose an integer linear program (ILP) that determines the optimal placement of experts, minimizing the expected number of transmissions.<n>We demonstrate that ILP-based placement strategy yields lower network traffic than competitors for small-scale (DeepSeekMoE16B) and large-scale (DeepSeek-R1671B) models.
  arXiv  Detail & Related papers  (2025-08-12T07:08:48Z)
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [65.18157595903124]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs [86.76714527437383]
  This paper proposes DSMoE, a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks.<n>We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge.<n>Experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches.
  arXiv  Detail & Related papers  (2025-02-18T02:37:26Z)
• MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing [0.6445605125467574]
  Mixture-of-Experts (MoE) model architecture has emerged as a promising solution for scaling transformer models efficiently.<n>MoE models need to be distributed across GPU devices, thus face critical performance bottlenecks.<n>We propose an optimal expert-to- GPU assignment that minimizes token routing costs and token processing balances across devices.
  arXiv  Detail & Related papers  (2025-02-10T16:34:36Z)
• HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference [54.40808356999408]
  We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency. HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
  arXiv  Detail & Related papers  (2024-11-03T04:25:46Z)
• AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference [13.263938935671646]
  AdapMoE is an algorithm-system co-design framework for efficient MoE inference. AdapMoE features adaptive expert gating and management to reduce the on-demand loading overheads. We show AdapMoE consistently outperforms existing techniques, reducing the average number of activated experts by 25% and achieving a 1.35x speedup without degradation accuracy.
  arXiv  Detail & Related papers  (2024-08-19T03:27:15Z)
• FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models [50.331708897857574]
  We introduce FactorLLM, a novel approach that decomposes well-trained dense FFNs into sparse sub-networks without requiring any further modifications. FactorLLM achieves comparable performance to the source model securing up to 85% model performance while obtaining over a 30% increase in inference speed.
  arXiv  Detail & Related papers  (2024-08-15T16:45:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.