SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding

• URL: http://arxiv.org/abs/2605.10453v1
• Date: Mon, 11 May 2026 12:22:37 GMT
• Title: SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
• Abstract summary: We propose SlimSpec, a low-rank parameterization of the drafter's LM-head that compresses the inner representation rather than the output.<n>SlimSpec achieves $4text-5times$ acceleration over the standard LM-head architecture.
• Score: 41.803148916760804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding speeds up autoregressive generation in Large Language Models (LLMs) through a two-step procedure, where a lightweight draft model proposes tokens which the target model then verifies in a single forward pass. Although the drafter network is small in modern architectures, its LM-head still performs projection to a large vocabulary, becoming one of the major computational bottlenecks. In prior work this issue has been predominantly addressed via static or dynamic vocabulary truncation. Yet mitigating the bottleneck, these methods bring in extra complexity, such as special vocabulary curation, sophisticated inference-time logic or modifications of the training setup. In this paper, we propose SlimSpec, a low-rank parameterization of the drafter's LM-head that compresses the inner representation rather than the output, preserving full vocabulary support. We evaluate our method with EAGLE-3 drafter across three target models and diverse benchmarks in both latency- and throughput-bound inference regimes. SlimSpec achieves $4\text{-}5\times$ acceleration over the standard LM-head architecture while maintaining a competitive acceptance length, surpassing existing methods by up to $8\text{-}9\%$ of the end-to-end speedup. Our method requires minimal adjustments of training and inference pipelines. Combined with the aforementioned speedup improvements, it makes SlimSpec a strong alternative across wide variety of draft LM-head architectures.

Related papers

• Unlocking Lossless Speedups in LLMs via Discrete Diffusion [67.4336730000437]
  Large Language Models (LLMs) owe much of their success to next-token prediction (NTP)<n>We introduce diffusion-augmented LLMs, a new class of models that defines an AR model distribution.<n>We decouple the parameters of these models into two sets: AR weights, trained using the standard NTP objective, and lightweight diffusion weights, trained to generate multiple tokens simultaneously.
  arXiv  Detail & Related papers  (2026-09-03T15:48:43Z)
• EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation [15.94674618023941]
  EvoSpec is a framework that enables real-time evolution of the draft model through dynamic vocabulary and parameter adaptation.<n>It achieves a 1.13x speedup in these settings over the state-of-the-art static baseline FR-Spec, with 27% lower memory overhead than standard online adaptation.
  arXiv  Detail & Related papers  (2026-04-17T06:16:58Z)
• Speculative Decoding with a Speculative Vocabulary [44.656073829954636]
  Speculative decoding is a leading approach for accelerating language model (LM) inference.<n>Recent work has sought to address this output distribution bottleneck by reducing the vocabulary of the draft model.<n>We propose SpecVocab, an efficient and effective method that selects a vocabulary subset per decoding step.
  arXiv  Detail & Related papers  (2026-02-14T16:10:00Z)
• CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs [48.8252978488871]
  Speculative decoding offers seamless integration and broad utility when deploying large language models.<n>Cascading a hierarchy of draft models promises further acceleration and flexibility, but the high cost of training multiple models has limited its practical application.<n>We propose a novel Cascade Adaptive Self-Speculative Decoding (CAS-Spec) method which constructs speculative draft models.
  arXiv  Detail & Related papers  (2025-10-30T08:51:29Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs [15.508475101753715]
  We introduce a training-free technique to improve the performance of drafter-based speculative decoding (SpD) methods.<n>A drafter-based speculative decoding leverages one or more smaller language models, a.k.a. drafters or draft models, to sample a draft sequence or tree consisting of multiple tokens.<n>We show that our method can boost the memory-bound speed-up for Llama-3 models on Spec-Bench, specifically by 16% for Llama-3.2-3B-Instruct.
  arXiv  Detail & Related papers  (2025-06-28T00:26:40Z)
• FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling [59.8051705468084]
  Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models.<n>We present FR-Spec, a frequency-ranked speculative sampling framework that optimize draft candidate selection through vocabulary space compression.
  arXiv  Detail & Related papers  (2025-02-20T18:58:10Z)
• Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree [7.438117410146904]
  Falcon is an innovative speculative decoding framework fashioned to augment both the drafter's parallelism and output quality.<n>Falcon incorporates the Coupled Sequential Glancing Distillation technique, which fortifies inter-token dependencies within the same block, leading to increased speculation accuracy.
  arXiv  Detail & Related papers  (2024-12-17T08:02:08Z)
• Extreme Compression of Large Language Models via Additive Quantization [59.3122859349777]
  Our algorithm, called AQLM, generalizes the classic Additive Quantization (AQ) approach for information retrieval. We provide fast GPU and CPU implementations of AQLM for token generation, which enable us to match or outperform optimized FP16 implementations for speed.
  arXiv  Detail & Related papers  (2024-01-11T18:54:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.