SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading

• URL: http://arxiv.org/abs/2609.12978v1
• Date: Fri, 11 Sep 2026 15:43:52 GMT
• Title: SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
• Abstract summary: Mixture-of-Experts (MoE) creates a structural advantage for offloading.<n>With 45% expert residency, SeqMoE averages a 96.97% hit rate and 80.22% of full-load performance.
• Score: 21.33671694736238
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) creates a structural advantage for offloading: only a small fraction of activated experts need to reside in device memory, and if they can be loaded in time for computation, offloading can in principle approach full-load performance, where all model weights reside in device memory. Yet translating MoE's structural advantage into practical offloading gains remains challenging. We propose SeqMoE to bridge this gap. To maximize expert hits, we build predictive memory management: (i) Sequence-to-sequence prediction. We are the first to recast expert activation prediction as sequence modeling, enabling accurate multi-step, multi-layer forecasts that provide a long and reliable window for downstream decisions. (ii) Joint prefetch scheduling. We formulate prefetch scheduling as Job Sequencing with Deadlines to maximize expected expert hits and improve bandwidth efficiency. (iii) Forecast-driven caching. Leveraging the recursive nature of sequence modeling, we introduce a probabilistic Belady policy for future-aware eviction. To eliminate execution bottleneck, we develop (iv) Graph-compatible offloading runtime. We derive general runtime principles encompassing compute-transparent expert placement and synchronization-free orchestration disciplines for end-to-end graph capture. With 45% expert residency, SeqMoE averages a 96.97% hit rate and 80.22% of full-load performance, advancing the state of the art in MoE offloading.

Related papers

• SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models [7.204022394734952]
  SpecPrefetch is a parameter-efficient prefetching framework for offloaded MoE inference.<n>It reduces exposed expert-loading latency without changing pretrained routing semantics.<n>On a Snapdragon 8 Elite device, SpecPrefetch further improves decoding throughput by up to (20%) over a compute-optimized offloading runtime.
  arXiv  Detail & Related papers  (2026-06-24T04:53:00Z)
• A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference [10.085075886001665]
  Mixture-of-Experts (MoE) based large language models (LLMs) have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost.<n>We propose ST-MoE, atemporal expert prefetching framework that proactively stages experts ahead of use to overlap expert loading with ongoing computation.
  arXiv  Detail & Related papers  (2026-06-13T20:09:54Z)
• SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning [104.01865949020304]
  Agentic multimodal large language models (MLLMs) achieve remarkable reasoning capabilities through iterative visual tool invocation.<n>However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead.<n>This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level throughput.<n>We propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck.
  arXiv  Detail & Related papers  (2026-03-24T17:45:47Z)
• Speculating Experts Accelerates Inference for Mixture-of-Experts [44.31811859704714]
  Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs)<n>We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts.<n>Our approach achieves up to 14% reduction in time per output token over on-demand loading of experts from CPU memory.
  arXiv  Detail & Related papers  (2026-03-09T06:59:47Z)
• Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts [74.40169987564724]
  Expert parallelism (EP) is designed to scale MoE models by distributing experts across multiple devices.<n>Under extreme imbalance, EP can funnel a disproportionate number of tokens to a small number of experts, leading to compute- and memory-bound failures.<n>We propose Least-Loaded Expert Parallelism (LLEP), a novel EP algorithm that dynamically reroutes excess tokens and associated expert parameters from overloaded devices to underutilized ones.
  arXiv  Detail & Related papers  (2026-01-23T18:19:15Z)
• Task-oriented Learnable Diffusion Timesteps for Universal Few-shot Learning of Dense Tasks [48.86985692711283]
  Current diffusion model-based applications exploit the power of learned visual representations from multistep forward-backward Markovian processes for single-task prediction tasks.<n>We propose two modules: Task-aware Timestep Selection (TTS) to select ideal diffusion timesteps based on timestep-wise losses and similarity scores, and Timestep Feature Consolidation (TFC) to consolidate the selected timestep features.<n>Our framework effectively achieves superiority in dense prediction performance given only a few support queries.
  arXiv  Detail & Related papers  (2025-12-29T05:19:01Z)
• MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts [29.437264687850874]
  We present MoE-SpeQ, a new inference system built on a novel co-design of speculative execution and expert offloading.<n>MoE-SpeQ employs a small, on-device draft model to predict the sequence of required experts for future tokens.<n>Our evaluation on memory-constrained devices shows that for the Phi-MoE model, MoE-SpeQ achieves at most 2.34x speedup over the state-of-the-art offloading framework.
  arXiv  Detail & Related papers  (2025-11-18T03:40:19Z)
• MC#: Mixture Compressor for Mixture-of-Experts Large Models [86.64315380917827]
  Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation.<n>We propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning.
  arXiv  Detail & Related papers  (2025-10-13T03:12:46Z)
• D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving [14.607254882119507]
  Combination of experts (MoE) model is a sparse variant of large language models (LLMs)<n>Despite its benefits, MoE is still too expensive to deploy on resource-constrained edge devices.<n>We propose D$2$MoE, an algorithm-system co-design framework that matches diverse task requirements by dynamically allocating the most proper bit-width to each expert.
  arXiv  Detail & Related papers  (2025-04-17T05:37:35Z)
• HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference [54.40808356999408]
  We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency. HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
  arXiv  Detail & Related papers  (2024-11-03T04:25:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.