MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

• URL: http://arxiv.org/abs/2602.16052v1
• Date: Tue, 17 Feb 2026 22:02:36 GMT
• Title: MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
• Abstract summary: We propose a training-free verification-time expert budgeting method that decouples speculation depth from memory cost.<n>Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines.
• Score: 4.064682810747041
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30\% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.

Related papers

• LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization [14.897282681862324]
  Speculative decoding accelerates large language model inference by drafting multiple tokens for parallel verification.<n>We develop textitLibraSpec, a training-free and plug-and-play algorithm that iteratively determines the speculative length using drafter confidence scores.
  arXiv  Detail & Related papers  (2026-08-09T14:15:14Z)
• Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts [34.20364644493512]
  We propose a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection.<n>We evaluate textscEcoSpec on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B.
  arXiv  Detail & Related papers  (2026-07-14T12:22:55Z)
• Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding [15.040000425985324]
  Tree-based speculative decoding accelerates autoregressive generation by verifying multiple draft candidates in parallel, but this advantage weakens for sparse Mixture-of-Experts (MoE) models.<n>We propose EVICT, a training-free, hyper parameter-free, and lossless adaptive verification method for MoE speculative decoding.<n> EVICT makes every verified token count by truncating the draft tree before target verification and retaining only the cost-effective prefix.
  arXiv  Detail & Related papers  (2026-05-01T01:52:01Z)
• SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning [104.01865949020304]
  Agentic multimodal large language models (MLLMs) achieve remarkable reasoning capabilities through iterative visual tool invocation.<n>However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead.<n>This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level throughput.<n>We propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck.
  arXiv  Detail & Related papers  (2026-03-24T17:45:47Z)
• TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification [63.65902785448346]
  Speculative decoding offers significant speed-ups through its lightweight drafting and parallel verification mechanism.<n>We propose TriSpec, a novel ternary SD framework that introduces a lightweight proxy to significantly reduce computational cost.<n>Experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35% speedup over standard SD.
  arXiv  Detail & Related papers  (2026-01-30T17:04:18Z)
• Accelerate Speculative Decoding with Sparse Computation in Verification [49.74839681322316]
  Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel.<n>Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding.<n>We propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost.
  arXiv  Detail & Related papers  (2025-12-26T07:53:41Z)
• SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification [11.366541829206199]
  Speculative decoding is one of the most direct and effective approaches for accelerating generation.<n>We introduce SpecPV, a self-speculative decoding approach that performs fast verification using partial key-value states.<n>We validate SpecPV across multiple long-context benchmarks and models, including LLaMA-3.1-8B-Instruct and Qwen3-series.
  arXiv  Detail & Related papers  (2025-12-02T02:15:33Z)
• Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining [34.97996720624093]
  We introduce a framework for dynamically re-routing token-to-expert mapping to lower MoE latency.<n>We evaluate our method on the Qwen3-30B and Qwen3-235B models with a batch size of $16$.
  arXiv  Detail & Related papers  (2025-11-04T04:00:20Z)
• HiSpec: Hierarchical Speculative Decoding for LLMs [15.347747465564458]
  We propose a framework for speculative decoding that exploits $textitearly-exit (EE) models for low-overhead intermediate verification.<n>We show that HiSpec improves throughput by 1.28$times$ on average and by up to 2.01$times$ compared to the baseline single-layer speculation.
  arXiv  Detail & Related papers  (2025-10-01T18:04:14Z)
• $\ exttt{SPECS}$: Faster Test-Time Scaling through Speculative Drafts [55.231201692232894]
  $textttSPECS$ is a latency-aware test-time scaling method inspired by speculative decoding.<n>Our results show that $textttSPECS$matches or surpasses beam search accuracy while reducing latency by up to $sim$19.1%.
  arXiv  Detail & Related papers  (2025-06-15T05:50:05Z)
• HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference [54.40808356999408]
  We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency. HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
  arXiv  Detail & Related papers  (2024-11-03T04:25:46Z)
• QSpec: Speculative Decoding with Complementary Quantization Schemes [53.960146187821685]
  Quantization is widely adopted to accelerate inference and reduce memory consumption in large language models (LLMs)<n>We propose QSpec, a novel quantization paradigm that decouples efficiency from quality.<n>QSpec reuses both weights and KV cache across stages, enabling near-zero-cost switching without retraining or auxiliary models.
  arXiv  Detail & Related papers  (2024-10-15T05:57:51Z)
• TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput [37.56866491624234]
  Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving.<n>We present TurboSpec, a speculation control system that automatically profiles the execution environment.<n>We demonstrate its effectiveness across diverse workloads and hardware configurations.
  arXiv  Detail & Related papers  (2024-06-20T07:43:33Z)
• Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization [51.98792406392873]
  Mixture of Experts (MoE) provides a powerful way to decompose dense layers into smaller, modular computations. A major challenge lies in the computational cost of scaling the number of experts high enough to achieve fine-grained specialization. We propose the Multilinear Mixture of Experts ($mu$MoE) layer to address this, focusing on vision models.
  arXiv  Detail & Related papers  (2024-02-19T21:20:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.