SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models

• URL: http://arxiv.org/abs/2607.24787v2
• Date: Thu, 30 Jul 2026 11:13:04 GMT
• Title: SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
• Abstract summary: SpecPrefetch is a parameter-efficient prefetching framework for offloaded MoE inference.<n>It reduces exposed expert-loading latency without changing pretrained routing semantics.<n>On a Snapdragon 8 Elite device, SpecPrefetch further improves decoding throughput by up to (20%) over a compute-optimized offloading runtime.
• Score: 7.204022394734952
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse Mixture-of-Experts (MoE) models expand foundation model capacity through conditional expert activation, but their full expert pools remain difficult to deploy under limited accelerator memory. Although expert offloading alleviates memory pressure by moving inactive experts to host memory or storage, it introduces a routing-dependent transfer bottleneck: required experts are known only after native top-\(K\) routing, which serializes routing, expert loading, and expert execution during inference. To address this bottleneck, we propose SpecPrefetch, a parameter-efficient prefetching framework for offloaded MoE inference. SpecPrefetch uses a shared lightweight adapter to predict next-layer expert candidates only for asynchronous transfer, while the frozen native router still determines the final executed experts. By separating transfer prediction from execution routing, SpecPrefetch reduces exposed expert-loading latency without changing pretrained routing semantics, so prediction errors affect transfer efficiency rather than model outputs. In addition, a window-aware scheduler prioritizes feasible transfers under cache and bandwidth constraints. Across Qwen3-VL-30B-A3B and DeepSeek-VL2-Tiny, SpecPrefetch achieves the best average expert recall in 9 out of 10 model-benchmark settings with substantially fewer trainable parameters than learned predictor baselines. On a Snapdragon 8 Elite device, SpecPrefetch further improves decoding throughput by up to \(20\%\) over a compute-optimized offloading runtime, demonstrating practical benefits for storage-constrained MoE deployment. The code and model weights are available at https://github.com/wei390/SpecPrefetch.

Related papers

• SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading [21.33671694736238]
  Mixture-of-Experts (MoE) creates a structural advantage for offloading.<n>With 45% expert residency, SeqMoE averages a 96.97% hit rate and 80.22% of full-load performance.
  arXiv  Detail & Related papers  (2026-09-11T15:43:52Z)
• Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts [49.09151538536423]
  Mixture-of-experts (MoE) models enable scalable transformer architectures by activating only a subset of experts per token.<n>Recent evidence suggests that performance improves with increasingly granular experts, i.e., many small experts instead of a few large ones.<n>We introduce adaptive inverted-index routing for MoE (AIR-MoE), an inverted-index-inspired routing architecture based on vector quantization (VQ)
  arXiv  Detail & Related papers  (2026-05-06T14:15:10Z)
• ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving [18.574823955774207]
  We propose ZeRO-Prefill, a prefill-only serving system whose backend gathers experts by weight rather than routing them by activation.<n>On Qwen3-235B-A22B across four hardware/precision configurations, ZeRO-Prefill delivers 1.35-1.37x throughput over the strongest distributed baseline on real-world workloads.
  arXiv  Detail & Related papers  (2026-05-03T03:10:24Z)
• Speculating Experts Accelerates Inference for Mixture-of-Experts [44.31811859704714]
  Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs)<n>We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts.<n>Our approach achieves up to 14% reduction in time per output token over on-demand loading of experts from CPU memory.
  arXiv  Detail & Related papers  (2026-03-09T06:59:47Z)
• Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts [74.40169987564724]
  Expert parallelism (EP) is designed to scale MoE models by distributing experts across multiple devices.<n>Under extreme imbalance, EP can funnel a disproportionate number of tokens to a small number of experts, leading to compute- and memory-bound failures.<n>We propose Least-Loaded Expert Parallelism (LLEP), a novel EP algorithm that dynamically reroutes excess tokens and associated expert parameters from overloaded devices to underutilized ones.
  arXiv  Detail & Related papers  (2026-01-23T18:19:15Z)
• BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference [11.5035097836611]
  Growing size of modern MoE models causes their full parameter sets to exceed GPU memory capacity.<n>Prefetchings aim to hide this latency by predicting which experts are needed, but prefetch failures introduce significant stalls and amplify inference latency.<n>The critical challenge is to maintain both high inference speed and model accuracy when prefetching fails.
  arXiv  Detail & Related papers  (2025-11-13T07:56:50Z)
• ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference [8.296993547783808]
  ExpertFlow is a runtime system for MoE inference that combines adaptive expert prefetching and cache-aware routing.<n>Our evaluation demonstrates that ExpertFlow reduces model stall time to less than 0.1% of the baseline.
  arXiv  Detail & Related papers  (2025-10-30T17:29:27Z)
• From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing [52.01745035243826]
  Mixture-of-Experts (MoE) models can scale parameter capacity by routing each token to a subset of experts.<n> conditional routing shifts the burden on inference memory, limiting the number of experts per device.<n>We present LASER, a plug-and-play, inference-time routing algorithm that balances load while preserving accuracy.
  arXiv  Detail & Related papers  (2025-09-29T16:29:17Z)
• Mixture of Lookup Experts [63.787712153454464]
  Mixture-of-Experts (MoE) activates only a subset of experts during inference.<n>MoLE is a new MoE architecture that is efficient in both communication and VRAM usage.
  arXiv  Detail & Related papers  (2025-03-20T02:31:57Z)
• HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference [54.40808356999408]
  We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency. HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
  arXiv  Detail & Related papers  (2024-11-03T04:25:46Z)
• ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference [41.41316718220569]
  ExpertFlow is designed to enhance inference efficiency by accommodating flexible routing and enabling efficient expert scheduling between CPU and GPU. Our experiments demonstrate that ExpertFlow achieves up to 93.72% GPU memory savings and enhances inference speed by 2 to 10 times compared to baseline methods.
  arXiv  Detail & Related papers  (2024-10-23T15:24:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.