SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths

• URL: http://arxiv.org/abs/2405.19715v3
• Date: Fri, 11 Jul 2025 01:33:18 GMT
• Title: SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths
• Abstract summary: Speculative decoding reduces the inference latency of a target large language model via utilizing a smaller and faster draft model.<n>We study the choice of the candidate length K and formulate it as a Markov Decision Process.<n>Motivated by this theory, we propose SpecDec++, an enhanced version of speculative decoding that adaptively determines the candidate length on the fly.
• Score: 41.863092939498316
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding reduces the inference latency of a target large language model via utilizing a smaller and faster draft model. Its performance depends on a hyperparameter K -- the candidate length, i.e., the number of candidate tokens for the target model to verify in each round. However, previous methods often use simple heuristics to choose K, which may result in sub-optimal performance. We study the choice of the candidate length K and formulate it as a Markov Decision Process. We theoretically show that the optimal policy of this Markov decision process takes the form of a threshold policy, i.e., the current speculation should stop and be verified when the probability of getting a rejection exceeds a threshold value. Motivated by this theory, we propose SpecDec++, an enhanced version of speculative decoding that adaptively determines the candidate length on the fly. We augment the draft model with a trained acceptance prediction head to predict the conditional acceptance probability of the candidate tokens. SpecDec++ will stop the current speculation when the predicted probability that at least one token gets rejected exceeds a threshold. We implement SpecDec++ and apply it to the llama-2-chat 7B & 70B model pair. Our adaptive method achieves a 2.04x speedup on the Alpaca dataset (7.2% improvement over the baseline speculative decoding). On the GSM8K and HumanEval datasets, our method achieves a 2.26x speedup (9.4% improvement) and 2.23x speedup (11.1% improvement), respectively. The code of this paper is available at https://github.com/Kaffaljidhmah2/SpecDec_pp.

Related papers

• ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping [79.1905186547363]
  ResiSpec is a framework that strategically reforms the proposal distribution during verification.<n>It prevents candidate obsolescence and achieves up to 1.92$times$ speedup over state-of-the-art multi-candidate methods.
  arXiv  Detail & Related papers  (2026-08-25T11:25:55Z)
• Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models [51.21958617138889]
  Self-Aware Scheduling learns a lightweight order policy and applies seamlessly to both any-order and semi-autoregressive decoding.<n>On Sudoku with 1B MDM, SAS improves puzzle accuracy from 82.0% (best schedule) to 91.8%, and reaches 97.5% with second-stage fine-tuning.<n>On mathematical reasoning with LLaDA-8B, SAS improves pass@1 on GSM8K from 64% to 76% and on MBPP from 39.5% to 41%.
  arXiv  Detail & Related papers  (2026-06-22T16:32:25Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• Acceptance Dynamics Across Cognitive Domains in Speculative Decoding [0.0]
  We present an empirical study of tree-based speculative decoding acceptance dynamics.<n>Our study spans four well-established NLP benchmark domains.<n>We find that task type is a stronger predictor of acceptance than tree depth.
  arXiv  Detail & Related papers  (2026-04-16T06:38:44Z)
• AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference [1.1852406625172216]
  We propose Adaptive Speculative Decoding (AdaSD) for large language models (LLMs)<n>AdaSD dynamically adjusts generation length and acceptance criteria during inference.<n> Experiments on benchmark datasets demonstrate that AdaSD achieves up to 49% speedup over standard speculative decoding.
  arXiv  Detail & Related papers  (2025-12-12T04:56:08Z)
• Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Minimization [1.2674961594128336]
  We devise an algorithm for optimal speculative sampling when the $n$ tokens are chosen from a single draft model.<n>Our findings give the first multi-draft algorithm with 90% acceptance and under 100 ms of overhead per generated token with negligible deviation from the target model distribution.
  arXiv  Detail & Related papers  (2025-11-19T21:59:43Z)
• Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning [85.76121000710522]
  We propose Re-FORC, an adaptive reward prediction method.<n>It enables prediction of the expected future rewards as a function of the number of future thinking tokens.
  arXiv  Detail & Related papers  (2025-11-03T23:47:49Z)
• Predictive Analytics for Collaborators Answers, Code Quality, and Dropout on Stack Overflow [5.4414562674321765]
  Previous studies that used Stack Overflow to develop predictive models often employed limited benchmarks of 3-5 models or adopted arbitrary selection methods.<n>Our study evaluates 21 algorithms across three tasks: predicting the number of question a user is likely to answer, their code quality violations, and their dropout status.
  arXiv  Detail & Related papers  (2025-06-23T06:23:12Z)
• One Sample is Enough to Make Conformal Prediction Robust [53.78604391939934]
  We show that conformal prediction attains some robustness even with a forward pass on a single randomly perturbed input.<n>Our approach returns robust sets with smaller average set size compared to SOTA methods which use many (e.g. around 100) passes per input.
  arXiv  Detail & Related papers  (2025-06-19T19:14:25Z)
• PMPO: Probabilistic Metric Prompt Optimization for Small and Large Language Models [1.6816171955882597]
  PMPO locates low quality prompt segments via a masking based analysis and iteratively rewrites them to propose improved variants.<n>It selects among variants by minimizing loss in a single forward pass, eliminating output sampling and human or judge based scoring for selection.<n>Across model sizes and datasets, PMPO outperforms prior prompts: it achieves the highest average accuracy on BBH, performs strongly on GSM8K and AQUA RAT, and raises AlpacaEval 2.0 win rates by over 19 points.
  arXiv  Detail & Related papers  (2025-05-22T06:59:10Z)
• AutoJudge: Judge Decoding Without Manual Annotation [10.411318392966358]
  AutoJudge is a framework that accelerates large language model (LLM) inference with task-specific lossy speculative decoding. We use a semi-greedy search algorithm to test which of the mismatches between target and draft model should be corrected. We then train a lightweight classifier based on existing LLM embeddings to predict, at inference time, which mismatching tokens can be safely accepted.
  arXiv  Detail & Related papers  (2025-04-28T17:59:28Z)
• Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding [0.0]
  Speculative decoding accelerates large language model inference.<n>We introduce textitGammaTune and textitGammaTune+, training-free adaptive algorithms that dynamically adjust speculation length based on token acceptance rates.
  arXiv  Detail & Related papers  (2025-03-28T23:41:55Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability. We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)
• Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization [66.67988187816185]
  We aim to emphscale up the number of on-policy samples via repeated random sampling to improve alignment performance. Our experiments reveal that this strategy leads to a emphdecline in performance as the sample size increases. We introduce a scalable preference data construction strategy that consistently enhances model performance as the sample scale increases.
  arXiv  Detail & Related papers  (2025-02-24T04:22:57Z)
• Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE [15.003006630308517]
  Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to predict multiple tokens. We propose Jakiro, leveraging Mixture of Experts (MoE), where independent experts generate diverse predictions. Our method significantly boosts prediction accuracy and achieves higher inference speedups.
  arXiv  Detail & Related papers  (2025-02-10T09:24:06Z)
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment [25.988070517700848]
  Speculative decoding has been proposed as a technique to accelerate autoregressive generation. We show that even powerful draft models such as GPT-4o, as well as human text cannot achieve high acceptance rates. We ask the following question: Can we adapt verification to recognize correct, but non-aligned replies?
  arXiv  Detail & Related papers  (2025-01-31T17:09:53Z)
• Iterative Reasoning Preference Optimization [84.15992372132507]
  We develop an iterative approach to optimize the preference between generated Chain-of-Thought (CoT) candidates. We show reasoning improves across repeated iterations of this scheme. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
  arXiv  Detail & Related papers  (2024-04-30T17:28:05Z)
• Decision-focused predictions via pessimistic bilevel optimization: a computational study [0.7499722271664147]
  Uncertainty in optimization parameters is an important and longstanding challenge. We build predictive models that measure a emphregret measure on decisions taken with them. We show various computational techniques to achieve tractability.
  arXiv  Detail & Related papers  (2023-12-29T15:05:00Z)
• Online Speculative Decoding [34.987825705622555]
  We introduce online speculative decoding to accelerate the inference of large language models. The main idea is to continuously update the (multiple) draft model(s) on observed user query data. We develop a prototype of online speculative decoding based on knowledge distillation and evaluate it using both synthetic and real query data.
  arXiv  Detail & Related papers  (2023-10-11T04:03:42Z)
• AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptation [64.9230895853942]
  Domain generalization can be arbitrarily hard without exploiting target domain information. Test-time adaptive (TTA) methods are proposed to address this issue. In this work, we adopt Non-Parametric to perform the test-time Adaptation (AdaNPC)
  arXiv  Detail & Related papers  (2023-04-25T04:23:13Z)
• Decoder Tuning: Efficient Language Understanding as Decoding [84.68266271483022]
  We present Decoder Tuning (DecT), which in contrast optimize task-specific decoder networks on the output side. By gradient-based optimization, DecT can be trained within several seconds and requires only one P query per sample. We conduct extensive natural language understanding experiments and show that DecT significantly outperforms state-of-the-art algorithms with a $200times$ speed-up.
  arXiv  Detail & Related papers  (2022-12-16T11:15:39Z)
• PyEPO: A PyTorch-based End-to-End Predict-then-Optimize Library for Linear and Integer Programming [9.764407462807588]
  We present the PyEPO package, a PyTorchbased end-to-end predict-then-optimize library in Python. PyEPO is the first such generic tool for linear and integer programming with predicted objective function coefficients.
  arXiv  Detail & Related papers  (2022-06-28T18:33:55Z)
• Multivariate Probabilistic Regression with Natural Gradient Boosting [63.58097881421937]
  We propose a Natural Gradient Boosting (NGBoost) approach based on nonparametrically modeling the conditional parameters of the multivariate predictive distribution. Our method is robust, works out-of-the-box without extensive tuning, is modular with respect to the assumed target distribution, and performs competitively in comparison to existing approaches.
  arXiv  Detail & Related papers  (2021-06-07T17:44:49Z)
• Non-Parametric Adaptive Network Pruning [125.4414216272874]
  We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
  arXiv  Detail & Related papers  (2021-01-20T06:18:38Z)
• Towards Improving Selective Prediction Ability of NLP Systems [24.774450633678125]
  We propose a method that improves probability estimates of models by calibrating them using prediction confidence and difficulty score of instances. We instantiate our method with Natural Language Inference (NLI) and Duplicate Detection (DD) tasks and evaluate it in both In-Domain (IID) and Out-of-Domain (OOD) settings.
  arXiv  Detail & Related papers  (2020-08-21T08:46:36Z)
• Regression with reject option and application to kNN [0.0]
  We refer to this framework as regression with reject option as an extension of classification with reject option. We provide a semi-supervised estimation procedure of the optimal rule involving two datasets. The resulting predictor with reject option is shown to be almost as good as the optimal predictor with reject option both in terms of risk and rejection rate.
  arXiv  Detail & Related papers  (2020-06-30T08:20:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.