DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

• URL: http://arxiv.org/abs/2607.24434v1
• Date: Mon, 27 Jul 2026 13:44:42 GMT
• Title: DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
• Abstract summary: DraftExpert is an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference.<n>On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU-GPU and Flash-NPU offload, DraftExpert improves decode throughput by 1.45x on average.
• Score: 0.4944564023471818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU. In this setting, self-speculative decoding faces a new bottleneck: increasing the draft expert set improves accuracy but triggers extra expert loading, while cheap small-footprint drafts have low acceptance; moreover, verifying a multi-token block activates the union of target experts and is no longer close to one target step. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. DraftExpert trains one lightweight accelerator-resident draft expert per layer by self-distilling residual, logit/token, and router-agreement signals from the frozen target MoE. At inference time, it uses a fixed-footprint shared+top-1+draft-expert drafter together with confidence--expansion truncation and target-expert prefetching, while final tokens are still exactly verified by the target model. On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU-GPU and Flash-NPU offload, DraftExpert improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.

Related papers

• ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration [28.21219412744595]
  Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute through expert activation.<n>We propose ExFold, jointly accelerating MoE prefill decode decode and batch-level expert-pool folding.
  arXiv  Detail & Related papers  (2026-08-24T08:53:58Z)
• AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding [21.76650680783374]
  Speculative decoding verifies a tree of draft tokens in one target-model forward pass.<n>For a mixture-of-experts (MoE) target, parallel verification can activate the union of the experts selected by all tree nodes.<n>We introduce AcceptMoE, a verifier-side expert selector that automatically adjusts the number of eligible experts for each verification block.
  arXiv  Detail & Related papers  (2026-08-04T01:01:12Z)
• Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts [34.20364644493512]
  We propose a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection.<n>We evaluate textscEcoSpec on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B.
  arXiv  Detail & Related papers  (2026-07-14T12:22:55Z)
• dMoE: dLLMs with Learnable Block Experts [71.572316901001]
  We propose dMoE, a simple yet effective block-level MoE framework.<n>We show that dMoE substantially reduces the number of uniquely activated experts during inference.<n>It also reduces memory usage by 76.64% to 79.84% and achieves 1.14$times$ to 1.66$times$ end-to-end latency speedup.
  arXiv  Detail & Related papers  (2026-05-29T06:03:50Z)
• Speculating Experts Accelerates Inference for Mixture-of-Experts [44.31811859704714]
  Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs)<n>We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts.<n>Our approach achieves up to 14% reduction in time per output token over on-demand loading of experts from CPU memory.
  arXiv  Detail & Related papers  (2026-03-09T06:59:47Z)
• MoE-Spec: Expert Budgeting for Efficient Speculative Decoding [4.064682810747041]
  We propose a training-free verification-time expert budgeting method that decouples speculation depth from memory cost.<n>Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines.
  arXiv  Detail & Related papers  (2026-02-17T22:02:36Z)
• BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference [11.5035097836611]
  Growing size of modern MoE models causes their full parameter sets to exceed GPU memory capacity.<n>Prefetchings aim to hide this latency by predicting which experts are needed, but prefetch failures introduce significant stalls and amplify inference latency.<n>The critical challenge is to maintain both high inference speed and model accuracy when prefetching fails.
  arXiv  Detail & Related papers  (2025-11-13T07:56:50Z)
• MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts [17.518573710849513]
  MoBiLE is a plug-and-play offloading-based MoE inference framework with textitmixture of big-little experts.<n>MoBiLE achieves a speedup of 1.60x to 1.72x compared to the baseline on a consumer GPU system, with negligible degradation in accuracy.
  arXiv  Detail & Related papers  (2025-10-14T10:22:44Z)
• MC#: Mixture Compressor for Mixture-of-Experts Large Models [86.64315380917827]
  Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation.<n>We propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning.
  arXiv  Detail & Related papers  (2025-10-13T03:12:46Z)
• Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding [73.67253077506672]
  Large language models (LLMs) deliver impressive generation quality, but incur very high inference cost.<n>Early-exit based self-speculative decoding (EESD) has emerged to mitigate this cost.<n>We propose Pipeline-Parallel Self-Speculative Decoding (PPSD) that fully pipelines the draft and verification work.
  arXiv  Detail & Related papers  (2025-09-19T04:51:41Z)
• ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts [79.62448915248926]
  Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing accuracy over the 16-bit model inference.<n>We propose using MXFP4 models as drafts in a plug-and-play fashion since the MXFP4 Weight-Only-Quantization (WOQ) merely direct-casts the BF16 target model weights to MXFP4.<n>In practice, our plug-and-play solution gives speedups up to 2x over the BF16 baseline.
  arXiv  Detail & Related papers  (2025-03-17T08:38:45Z)
• Bootstrapped Masked Autoencoders for Vision BERT Pretraining [142.5285802605117]
  BootMAE improves the original masked autoencoders (MAE) with two core designs. 1) momentum encoder that provides online feature as extra BERT prediction targets; 2) target-aware decoder that tries to reduce the pressure on the encoder to memorize target-specific information in BERT pretraining.
  arXiv  Detail & Related papers  (2022-07-14T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.