Reject Only Critical Tokens: Pivot-Aware Speculative Decoding

• URL: http://arxiv.org/abs/2511.00351v1
• Date: Sat, 01 Nov 2025 01:35:10 GMT
• Title: Reject Only Critical Tokens: Pivot-Aware Speculative Decoding
• Authors: Amir Ziashahabi, Yavuz Faruk Bakman, Duygu Nur Yaldiz, Mostafa El-Khamy, Sai Praneeth Karimireddy, Salman Avestimehr,
• Abstract summary: Speculative Decoding (SD) ensures that the output matches the target model's distribution exactly.<n>We propose Pivot-Aware Speculative Decoding, which rejects only those tokens that would lead to a utility drop in the final output.<n>We evaluate our method across various datasets, demonstrating that we can achieve up to $2.5times$ speedup with comparable utility.
• Score: 31.22793593647334
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative Decoding (SD) ensures that the output matches the target model's distribution exactly. However, we argue that this distribution matching requirement is too stringent and results in unnecessarily low acceptance rates, limiting potential speedups. Instead, we advocate a reformulation of the decoding objective: the proposed decoding strategy should match the expected utility, i.e., the task-specific performance, of the target model. This perspective also aligns better with real-world use cases of LLMs, where utility (e.g., code correctness, factual accuracy) is often more important than sampling distribution. Based on this reformulation, we propose a novel decoding strategy: Pivot-Aware Speculative Decoding, which rejects only those tokens that would lead to a utility drop in the final output. We refer to these critical tokens as pivot tokens. We propose a method for labeling tokens as pivotal or non-pivotal and train a lightweight classifier to detect them. This method can be viewed as a relaxed version of standard SD, which offers much higher acceptance while preserving utility. We evaluate our method across various datasets, demonstrating that we can achieve up to $2.5\times$ speedup with comparable utility. Source code is available at https://github.com/amir-zsh/PAD.

Related papers

• Residual Context Diffusion Language Models [90.07635240595926]
  Residual Context Diffusion (RCD) is a module that converts discarded token representations into contextual residuals and injects them back for the next denoising step.<n>RCD consistently improves frontier dLLMs by 5-10 points in accuracy with minimal extra computation overhead.
  arXiv  Detail & Related papers  (2026-01-30T13:16:32Z)
• dParallel: Learnable Parallel Decoding for dLLMs [77.24184219948337]
  Diffusion large language models (dLLMs) offer parallel token prediction and lower inference latency.<n>Existing open-source models still require nearly token-length decoding steps to ensure performance.<n>We introduce dParallel, a simple and effective method that unlocks the inherent parallelism of dLLMs for fast sampling.
  arXiv  Detail & Related papers  (2025-09-30T16:32:52Z)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding [60.407727995313074]
  We introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel.<n>APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-31T06:10:10Z)
• Foundations of Top-$k$ Decoding For Language Models [19.73575905188064]
  We develop a theoretical framework that both explains and generalizes top-$k$ decoding.<n>We show how to optimize it efficiently for a large class of divergences.
  arXiv  Detail & Related papers  (2025-05-25T23:46:34Z)
• Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding [48.52389201779425]
  Speculative decoding accelerates inference by generating multiple draft tokens using a lightweight model and verifying them in parallel.<n>Existing verification methods rely heavily on distributional consistency while overlooking semantic correctness.<n>We propose Reflective Verification, a training-free and semantics-aware approach that achieves a better trade-off between correctness and efficiency.
  arXiv  Detail & Related papers  (2025-05-24T10:26:27Z)
• AutoJudge: Judge Decoding Without Manual Annotation [13.451750613294054]
  AutoJudge is a method that accelerates large language model (LLM) inference with task-specific lossy speculative decoding.<n>Our approach relies on a semi-greedy search algorithm to test which of the mismatches between target and draft models should be corrected.
  arXiv  Detail & Related papers  (2025-04-28T17:59:28Z)
• EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models [40.651650382105636]
  Vanilla method adds padding tokens in order to ensure that the number of new tokens remains consistent across samples. We propose a novel method that can resolve the issue of inconsistent tokens accepted by different samples without necessitating an increase in memory or computing overhead. Our proposed method can handle the situation where the prediction tokens of different samples are inconsistent without the need to add padding tokens.
  arXiv  Detail & Related papers  (2024-05-13T08:24:21Z)
• Object Recognition as Next Token Prediction [99.40793702627396]
  We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels.
  arXiv  Detail & Related papers  (2023-12-04T18:58:40Z)
• Assessing Keyness using Permutation Tests [0.0]
  We replace the token-by-token sampling model by a model where corpora are samples of documents rather than tokens. We do not need any assumption on how the tokens are organized within or across documents, and the approach works with basically *any* keyness score.
  arXiv  Detail & Related papers  (2023-08-25T13:52:57Z)
• Coping with Label Shift via Distributionally Robust Optimisation [72.80971421083937]
  We propose a model that minimises an objective based on distributionally robust optimisation (DRO) We then design and analyse a gradient descent-proximal mirror ascent algorithm tailored for large-scale problems to optimise the proposed objective.
  arXiv  Detail & Related papers  (2020-10-23T08:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.