SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection

• URL: http://arxiv.org/abs/2605.02888v2
• Date: Tue, 05 May 2026 12:57:37 GMT
• Title: SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
• Abstract summary: SpecKV is a lightweight adaptive controller that selects $$ per speculation step using signals extracted from the draft model itself.<n>We demonstrate that the optimal $$ shifts across compression regimes and that draft model confidence and entropy are strong predictors of acceptance rate.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding accelerates large language model (LLM) inference by using a small draft model to propose candidate tokens that a larger target model verifies. A critical hyperparameter in this process is the speculation length $γ$, which determines how many tokens the draft model proposes per step. Nearly all existing systems use a fixed $γ$ (typically 4), yet empirical evidence suggests that the optimal value varies across task types and, crucially, depends on the compression level applied to the target model. In this paper, we present SpecKV, a lightweight adaptive controller that selects $γ$ per speculation step using signals extracted from the draft model itself. We profile speculative decoding across 4 task categories, 4 speculation lengths, and 3 compression levels (FP16, INT8, NF4), collecting 5,112 step-level records with per-step acceptance rates, draft entropy, and draft confidence. We demonstrate that the optimal $γ$ shifts across compression regimes and that draft model confidence and entropy are strong predictors of acceptance rate (correlation $\approx$ 0.56). SpecKV uses a small MLP trained on these signals to maximize expected tokens per speculation step, achieving a 56.0% improvement over the fixed-$γ=4$ baseline with only 0.34 ms overhead per decision (<0.5% of step time). The improvement is statistically significant (p < 0.001, paired bootstrap test). We release all profiling data, trained models, and notebooks as open-source artifacts.

Related papers

• Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware [0.0]
  Single-stream autoregressive decoding of large language models is bound by memory bandwidth.<n>A small draft model proposes $K$ tokens autoregressively, the target model scores all of them in one batched pass.<n>We present a from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations.
  arXiv  Detail & Related papers  (2026-07-19T15:01:35Z)
• When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models [2.064923532131528]
  Deployed language models must produce outputs that are both correct and format-compliant.<n>We study this structured-output reliability gap using two mathematical benchmarks -- GSM8K and MATH.<n>We develop AloLab, an iterative system-prompt requiring only black-box API access to the target model.
  arXiv  Detail & Related papers  (2026-05-04T09:07:44Z)
• Compute Optimal Tokenization [79.3815358070537]
  We investigate how the information granularity of tokens, controlled by the compression rate, affects scaling trends.<n>We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate.<n>Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived.
  arXiv  Detail & Related papers  (2026-05-02T01:53:22Z)
• Token-Efficient Multimodal Reasoning via Image Prompt Packaging [0.6465251961564605]
  We introduce Image Prompt Packaging, a prompting paradigm that embeds structured text directly into images to reduce text token overhead.<n>We benchmark it across five datasets, three frontier models, and two task families.<n>We derive a cost decomposing savings by token type and show IPPg achieves 35.8--91.0% inference cost reductions.
  arXiv  Detail & Related papers  (2026-04-02T19:50:59Z)
• Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Minimization [1.2674961594128336]
  We devise an algorithm for optimal speculative sampling when the $n$ tokens are chosen from a single draft model.<n>Our findings give the first multi-draft algorithm with 90% acceptance and under 100 ms of overhead per generated token with negligible deviation from the target model distribution.
  arXiv  Detail & Related papers  (2025-11-19T21:59:43Z)
• CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning [62.56541355300587]
  We introduce a general test-time calibration framework that adaptively modifies the model toward high-reward reasoning paths.<n>Within this framework, we propose CarBoN, a two-phase method that first explores the solution space and then learns a calibration of the logits.<n>Experiments on MATH-500 and AIME-2024 show that CarBoN improves efficiency, with up to $4times$ fewer rollouts to reach the same accuracy.
  arXiv  Detail & Related papers  (2025-10-17T14:04:37Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• P$^2$U: Progressive Precision Update For Efficient Model Distribution [2.3349787245442966]
  We propose Progressive Precision Update (P$2$U) to address this problem.<n>Instead of transmitting the original high-precision model, P$2$U transmits a lower-bit precision model.<n>P$2$U consistently achieves better tradeoff between accuracy, bandwidth usage and latency.
  arXiv  Detail & Related papers  (2025-06-28T12:47:04Z)
• ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts [79.62448915248926]
  Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing accuracy over the 16-bit model inference.<n>We propose using MXFP4 models as drafts in a plug-and-play fashion since the MXFP4 Weight-Only-Quantization (WOQ) merely direct-casts the BF16 target model weights to MXFP4.<n>In practice, our plug-and-play solution gives speedups up to 2x over the BF16 baseline.
  arXiv  Detail & Related papers  (2025-03-17T08:38:45Z)
• Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment [6.947361774195549]
  We propose a modular classification pipeline that divides the relevance assessment task into multiple stages.<n>One of our approaches showed an 18.4% Krippendorff's $alpha$ accuracy increase over OpenAI's GPT-4o mini.
  arXiv  Detail & Related papers  (2025-01-24T07:33:39Z)
• FABind: Fast and Accurate Protein-Ligand Binding [127.7790493202716]
  $mathbfFABind$ is an end-to-end model that combines pocket prediction and docking to achieve accurate and fast protein-ligand binding. Our proposed model demonstrates strong advantages in terms of effectiveness and efficiency compared to existing methods.
  arXiv  Detail & Related papers  (2023-10-10T16:39:47Z)
• Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens [65.4435926060951]
  We propose to significantly improve the efficiency of Transformers for ultra long sequences, by compressing the sequence into a much smaller representation at each layer. Our algorithm is not only efficient (achieving more than $3times$ efficiency gain compared to baselines on 4K and 16K lengths) but also offers competitive/better performance on a large number of tasks.
  arXiv  Detail & Related papers  (2023-05-07T10:32:18Z)
• The Right Tool for the Job: Matching Model and Instance Complexities [62.95183777679024]
  As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. We propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) "exit" We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks.
  arXiv  Detail & Related papers  (2020-04-16T04:28:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.