ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

• URL: http://arxiv.org/abs/2605.02960v1
• Date: Sun, 03 May 2026 03:10:24 GMT
• Title: ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
• Abstract summary: We propose ZeRO-Prefill, a prefill-only serving system whose backend gathers experts by weight rather than routing them by activation.<n>On Qwen3-235B-A22B across four hardware/precision configurations, ZeRO-Prefill delivers 1.35-1.37x throughput over the strongest distributed baseline on real-world workloads.
• Score: 18.574823955774207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose ZeRO-Prefill, a prefill-only serving system whose backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than routing them by activation, and whose frontend co-enforces a physically-derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, ZeRO-Prefill delivers 1.35-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.

Related papers

• SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models [7.204022394734952]
  SpecPrefetch is a parameter-efficient prefetching framework for offloaded MoE inference.<n>It reduces exposed expert-loading latency without changing pretrained routing semantics.<n>On a Snapdragon 8 Elite device, SpecPrefetch further improves decoding throughput by up to (20%) over a compute-optimized offloading runtime.
  arXiv  Detail & Related papers  (2026-06-24T04:53:00Z)
• MOSAIC: Efficient Mixture-of-Agent Scheduling via Adaptive Aggregation and Inference Concurrency [11.75073577935001]
  Mixture-of-Agents (MoA) systems improve reasoning accuracy by routing each query to multiple expert LLMs and aggregating their outputs.<n>Traditional scheduling strategies suffer from significant GPU idling and throughput collapse due to load imbalances.<n>We present MOSAIC, a scheduling framework to accelerate MoA workloads on limited GPU resources.
  arXiv  Detail & Related papers  (2026-06-02T01:40:33Z)
• How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving [14.552785121277529]
  We study the benefits and limits of Attention-FFN Disaggregation (AFD) at scale through a framework that fuses on-device kernel measurements with high-fidelity network simulation.<n>Under strict TTFT/TPOT SLOs, AFD sustains around 4k tokens/s of system throughput on DeepSeek-V3.2 across chat, coding, and agentic-coding workloads.
  arXiv  Detail & Related papers  (2026-05-27T10:55:57Z)
• SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference [11.392365535797586]
  LLM serving platforms are increasingly deployed as multi-model cloud systems, where user demand is often long-tailed.<n>We propose SPECTRE, a framework that reuses underutilized tail-model services as remote drafters for heavily loaded large-model services.<n>We show that SPECTRE consistently improves large-model serving throughput while causing only minor interference to the native workloads of tail-model services.
  arXiv  Detail & Related papers  (2026-05-04T01:27:45Z)
• SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning [104.01865949020304]
  Agentic multimodal large language models (MLLMs) achieve remarkable reasoning capabilities through iterative visual tool invocation.<n>However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead.<n>This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level throughput.<n>We propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck.
  arXiv  Detail & Related papers  (2026-03-24T17:45:47Z)
• Stacked from One: Multi-Scale Self-Injection for Context Window Extension [69.24689919827817]
  modelname is a novel framework based on multi-grained context compression and query-aware information acquisition.<n>modelnameachieves performance superior or comparable to strong baselines.
  arXiv  Detail & Related papers  (2026-03-05T03:16:16Z)
• Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts [74.40169987564724]
  Expert parallelism (EP) is designed to scale MoE models by distributing experts across multiple devices.<n>Under extreme imbalance, EP can funnel a disproportionate number of tokens to a small number of experts, leading to compute- and memory-bound failures.<n>We propose Least-Loaded Expert Parallelism (LLEP), a novel EP algorithm that dynamically reroutes excess tokens and associated expert parameters from overloaded devices to underutilized ones.
  arXiv  Detail & Related papers  (2026-01-23T18:19:15Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs [61.953548065938385]
  We introduce the ''Three Taxes'' (Bulk Synchronous, Inter- Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework.<n>We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution.<n>We observe a 10-20% speedup in end-to-end latency over BSP-based approaches.
  arXiv  Detail & Related papers  (2025-11-04T01:15:44Z)
• Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving [4.309392302169281]
  Engine-level prefill-decode (PD) disaggregation avoids interference but incurs higher hardware and coordination overhead.<n>PD achieves up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT than vLLM; outperforms SG by up to 2x; and matches or exceeds disaggregated vLLM.
  arXiv  Detail & Related papers  (2025-07-09T07:27:18Z)
• D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving [14.607254882119507]
  Combination of experts (MoE) model is a sparse variant of large language models (LLMs)<n>Despite its benefits, MoE is still too expensive to deploy on resource-constrained edge devices.<n>We propose D$2$MoE, an algorithm-system co-design framework that matches diverse task requirements by dynamically allocating the most proper bit-width to each expert.
  arXiv  Detail & Related papers  (2025-04-17T05:37:35Z)
• BAFFLE: A Baseline of Backpropagation-Free Federated Learning [71.09425114547055]
  Federated learning (FL) is a general principle for decentralized clients to train a server model collectively without sharing local data. We develop backpropagation-free federated learning, dubbed BAFFLE, in which backpropagation is replaced by multiple forward processes to estimate gradients. BAFFLE is 1) memory-efficient and easily fits uploading bandwidth; 2) compatible with inference-only hardware optimization and model quantization or pruning; and 3) well-suited to trusted execution environments.
  arXiv  Detail & Related papers  (2023-01-28T13:34:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.