Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation

• URL: http://arxiv.org/abs/2411.18462v2
• Date: Sun, 24 Aug 2025 15:48:28 GMT
• Title: Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation
• Authors: Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, Zhaopeng Tu,
• Abstract summary: Experimental results on mainstream SD benchmarks as well as reasoning-heavy benchmarks demonstrate the superior performance of SVIP.<n>SVIP is a training-free dynamic length policy for speculative decoding systems that adaptively determines the lengths of draft sequences by referring to the draft entropy.
• Score: 64.59292053188264
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Conventional speculative decoding (SD) methods utilize a predefined length policy for proposing drafts, which implies the premise that the target model smoothly accepts the proposed draft tokens. However, reality deviates from this assumption: the oracle draft length varies significantly, and the fixed-length policy hardly satisfies such a requirement. Moreover, such discrepancy is further exacerbated in scenarios involving complex reasoning and long-form generation, particularly under test-time scaling for reasoning-specialized models. Through both theoretical and empirical estimation, we establish that the discrepancy between the draft and target models can be approximated by the draft model's prediction entropy: a high entropy indicates a low acceptance rate of draft tokens, and vice versa. Based on this insight, we propose SVIP: Self-Verification Length Policy for Long-Context Speculative Decoding, which is a training-free dynamic length policy for speculative decoding systems that adaptively determines the lengths of draft sequences by referring to the draft entropy. Experimental results on mainstream SD benchmarks as well as reasoning-heavy benchmarks demonstrate the superior performance of SVIP, achieving up to 17% speedup on MT-Bench at 8K context compared with fixed draft lengths, and 22% speedup for QwQ in long-form reasoning.

Related papers

• PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length [21.738896310075678]
  Speculative decoding (SD) is a powerful technique for accelerating the inference process of large language models (LLMs)<n>We propose Pacer, a novel approach that dynamically controls draft length using a lightweight, trainable pre-verification layer.<n>Our results demonstrate that Pacer achieves up to 2.66x Speedup over autoregressive decoding and consistently outperforms standard speculative decoding.
  arXiv  Detail & Related papers  (2026-02-01T15:12:38Z)
• AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference [1.1852406625172216]
  We propose Adaptive Speculative Decoding (AdaSD) for large language models (LLMs)<n>AdaSD dynamically adjusts generation length and acceptance criteria during inference.<n> Experiments on benchmark datasets demonstrate that AdaSD achieves up to 49% speedup over standard speculative decoding.
  arXiv  Detail & Related papers  (2025-12-12T04:56:08Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• Efficient Pretraining Length Scaling [21.4715211093876]
  We present the Parallel Hidden Decoding Transformer (textitPHD-Transformer), a novel framework that enables efficient length scaling during pre-training. textitPHD-Transformer achieves this through an innovative KV cache management strategy that distinguishes between original tokens and hidden decoding tokens.
  arXiv  Detail & Related papers  (2025-04-21T09:41:26Z)
• DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding [7.204881999658682]
  We introduce DEL, a plug-and-play method that adaptively selects the exit layer and speculation length during inference. Del achieves overall speedups of $2.16times$$sim$$2.50times$ over vanilla auto-regressive decoding.
  arXiv  Detail & Related papers  (2025-04-08T01:12:59Z)
• DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting [59.57151419673759]
  Speculative decoding presents a draft-then-verify framework that reduces generation latency while maintaining output distribution fidelity. We propose DuoDecoding, a novel approach that strategically deploys the draft and target models on the CPU and GPU respectively. Our method incorporates a hardware-aware optimal draft budget to minimize idle times and employs dynamic multi-sequence drafting to enhance draft quality.
  arXiv  Detail & Related papers  (2025-03-02T08:27:48Z)
• GRIFFIN: Effective Token Alignment for Faster Speculative Decoding [52.905060461479856]
  GRIFFIN is a framework that incorporates a token-alignable training strategy and a token-alignable draft model. Experiments on LLaMA-series and Vicuna models demonstrate that GRIFFIN achieves an average acceptance length improvement of over 7% and a speedup ratio exceeding 8%.
  arXiv  Detail & Related papers  (2025-02-16T07:06:00Z)
• AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures [11.436315332919245]
  We introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures. AdaEAGLE achieves a $1.62times$ speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline.
  arXiv  Detail & Related papers  (2024-12-25T13:57:33Z)
• Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration [14.011702040133848]
  We propose a CTC-based draft model which strengthens the correlations between draft tokens during the draft phase.<n>Experiment results show that compared to strong baselines, the proposed method can achieve a higher acceptance rate and hence a faster inference speed.
  arXiv  Detail & Related papers  (2024-11-25T14:10:21Z)
• What is Wrong with Perplexity for Long-context Language Modeling? [71.34933096461124]
  Long-context inputs are crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning.<n>Perplexity (PPL) has proven unreliable for assessing long-context capabilities.<n>We propose bfLongPPL, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them.
  arXiv  Detail & Related papers  (2024-10-31T09:39:28Z)
• AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability [5.421949344085942]
  We show that AdaEDL consistently outperforms static draft-length speculative decoding by 10%-57%. We also show that AdaEDL is more robust than these techniques and preserves performance in high-temperature scenarios.
  arXiv  Detail & Related papers  (2024-10-24T01:13:43Z)
• Improving Multi-candidate Speculative Decoding [1.6291177798903276]
  Speculative Decoding (SD) is a technique to accelerate the inference of Large Language Models (LLMs) In this work, we introduce a new version of MCSD that includes a target model multi-candidate generation. We also evaluate the effects of using the target model multi-candidate process with different draft models on output quality.
  arXiv  Detail & Related papers  (2024-09-16T18:20:38Z)
• Parallel Speculative Decoding with Adaptive Draft Length [10.36819001596531]
  We propose a conceptually simple, flexible, and general framework to boost speculative decoding. PEARL proposes textitpre-verify to verify the first draft token in advance during the drafting phase, and textitpost-verify to generate more draft tokens during the verification phase. PEARL parallels the drafting phase and the verification phase via applying the two strategies, and achieves adaptive draft length for different scenarios.
  arXiv  Detail & Related papers  (2024-08-13T08:32:06Z)
• Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion [59.17158389902231]
  Speculative decoding has emerged as a widely adopted method to accelerate large language model inference. This paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences.
  arXiv  Detail & Related papers  (2024-08-10T21:24:25Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism [35.7077090639665]
  We propose a novel approach called Early-exiting Speculative Decoding (EESD) with lossless acceleration. EESD utilizes a segment of the large language models (LLMs) to generate draft tokens, incorporating Early-exiting structures after the first N layers. We show that our approach decodes tokens at a markedly accelerated rate compared to prior methods.
  arXiv  Detail & Related papers  (2024-06-06T08:40:28Z)
• Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution [110.99891169486366]
  We propose a method that integrates efficient and precise uncertainty quantification into a deep learning-based surrogate model. Our method endows deep learning-based surrogate models with robust and efficient uncertainty quantification capabilities for both forward and inverse problems. Our method excels at propagating uncertainty over extended auto-regressive rollouts, making it suitable for scenarios involving long-term predictions.
  arXiv  Detail & Related papers  (2024-02-13T11:22:59Z)
• Revisiting the Power of Prompt for Visual Tuning [50.11465784194896]
  This study explores the correlation evolvement between prompts and patch tokens during proficient training. Inspired by the observation that the prompt tokens tend to share high mutual information with patch tokens, we propose initializing prompts with downstream token prototypes. Our method significantly advances the adaptation for self-supervised pretraining, achieving impressive task performance gains of at least 10% to 30%.
  arXiv  Detail & Related papers  (2024-02-04T07:49:02Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• DistillSpec: Improving Speculative Decoding via Knowledge Distillation [70.61777015900272]
  Speculative decoding (SD) accelerates large language model inference by employing a faster draft model for generating multiple tokens. We propose DistillSpec that uses knowledge distillation to better align the draft model with the target model, before applying SD. We show that DistillSpec yields impressive 10 - 45% speedups over standard SD on a range of standard benchmarks.
  arXiv  Detail & Related papers  (2023-10-12T16:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.