Approximate Speculative Decoding

• URL: http://arxiv.org/abs/2608.03447v1
• Date: Tue, 04 Aug 2026 10:45:24 GMT
• Title: Approximate Speculative Decoding
• Abstract summary: Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel.<n>We introduce textbf Speculative Decoding (ASD), a training-free verifier that replaces binary first-mismatch truncation with budgeted longest selection.<n>ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero.
• Score: 2.7677713446315946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce \textbf{Approximate Speculative Decoding (ASD)}, a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by $3.05\%$--$15.26\%$ over matched strict verification and averages a $7.78\%$ gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly $10\%$--$16\%$ on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: https://github.com/Kissmetothemoon/ASD

Related papers

• HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation [56.14416807766864]
  Homomorphic encryption allows a server to run a language model directly on encrypted user prompts.<n>Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance.<n>We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonline iteration counts learnable.
  arXiv  Detail & Related papers  (2026-09-01T18:02:02Z)
• Verification-Aware Training for Speculative Decoding [57.84976863792784]
  We introduce Verification-Aware Training (VAT), a plug-in framework that simulates verification at every training step and turns the resulting accept and reject patterns into supervision.<n>VAT modifies only the training objective, so it can be layered on top of existing methods without changing the draft architecture, the target model, or the inference procedure.<n>VAT improves average acceptance length by up to 11.4% and wall-clock speedup by up to 8.7%, with consistent gains across math, code, and chat benchmarks.
  arXiv  Detail & Related papers  (2026-08-31T01:42:10Z)
• From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding [41.41611499810411]
  Speculative diffusion decoding (SDD) generates every position in a draft block in parallel with a discrete diffusion model.<n>This paper recognizes that this creates a new control handle: whether to invoke the verifier at all.
  arXiv  Detail & Related papers  (2026-08-14T18:00:08Z)
• Fixed-order postselected CHSH reference acquisition for parity-constrained spatial-mode qubits on a commercial cloud photonic processor [0.0]
  We report a fixed-order Clauser-Horne-Shimony-Holt (CHSH) acquisition and reporting protocol for two encoded photonic qubits on Quandela's commercial, cloud-accessible Belenos processor.<n>Each logical qubit is a two-dimensional spatial-mode subspace in the seven-dimensional zero-sum sector of an eight-mode single-photon register.
  arXiv  Detail & Related papers  (2026-08-13T01:08:02Z)
• Ripple-Pivot Search: Active Parallel Decoding for Diffusion Large Language Models [66.32132912853372]
  Diffusion Large Language Models (dLLMs) have emerged as a competitive alternative to autoregressive language models.<n>We propose Ripple-Pivot Search (RPS), a training-free decoding method that seeks mid-entropy positions as promising candidate pivots.<n>RPS achieves 4-10$times$ wall-clock speedup over the standard decoder while preserving generation quality, and improves accuracy over the previous lookahead baseline by up to 5.49%.
  arXiv  Detail & Related papers  (2026-08-12T07:32:29Z)
• Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware [0.0]
  Single-stream autoregressive decoding of large language models is bound by memory bandwidth.<n>A small draft model proposes $K$ tokens autoregressively, the target model scores all of them in one batched pass.<n>We present a from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations.
  arXiv  Detail & Related papers  (2026-07-19T15:01:35Z)
• Self-Evolving Agents with Anytime-Valid Certificates [1.2691047660244335]
  We present textbfSEA, an architecture that confines self-modification to a small steering adapter and a versioned harness around a emphfrozen base model.<n>Five loop controllers compose published guarantees; because such gates can only emphselect among behaviors the frozen base already produces, five verifier-in-the-loop mechanisms supply the dense, grader-free signal the gates require.<n>Results are single-run on expensive evaluations; confirming run-to-run variance and adapting the per-task algorithm mix are future work.
  arXiv  Detail & Related papers  (2026-07-01T12:34:52Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• Minimal-Intervention KV Retention via Set-Conditioned Diversity [7.208745673318648]
  KV-cache compression at small budgets is a crowded design space spanning cache representation, head-wise routing, compression cadence, decoding behavior, and within-budget scoring.<n>We propose a one-function modification to the TriAttentioncitemao2026triattention retention scorer that replaces argmax-top-$k$ with greedy facility-location-inspired selection.<n>A pre-registered protocol tunes $$ on a frozen development split and confirms on a disjoint held-out split; with $= 0.5$, $$ clears Bonferroni on two of the four (
  arXiv  Detail & Related papers  (2026-05-14T02:50:20Z)
• Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editing [2.6382975801439836]
  LLMs edit text and code by autoregressively regenerating the full output, even when most tokens appear verbatim in the input.<n>We study Copy-as-Decode, a decoding-layer mechanism that recasts edit generation as structured decoding over a two-primitive grammar.
  arXiv  Detail & Related papers  (2026-04-20T12:29:53Z)
• RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding [80.12789199134511]
  Autoregressive decoding in Large Language Models (LLMs) generates one token per step, causing high inference latency.<n>We propose $textbfRACER, a lightweight and training-free method that integrates retrieved exact patterns with logit-driven future cues.<n>Experiments on Spec-Bench, HumanEval, and MGSM-ZH demonstrate that RACER consistently accelerates inference, achieving more than $2times$ speedup over autoregressive decoding.
  arXiv  Detail & Related papers  (2026-04-16T11:23:55Z)
• Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding [73.67253077506672]
  Large language models (LLMs) deliver impressive generation quality, but incur very high inference cost.<n>Early-exit based self-speculative decoding (EESD) has emerged to mitigate this cost.<n>We propose Pipeline-Parallel Self-Speculative Decoding (PPSD) that fully pipelines the draft and verification work.
  arXiv  Detail & Related papers  (2025-09-19T04:51:41Z)
• Estimating the Decoding Failure Rate of Binary Regular Codes Using Iterative Decoding [84.0257274213152]
  We propose a new technique to provide accurate estimates of the DFR of a two-iterations (parallel) bit flipping decoder.<n>We validate our results, providing comparisons of the modeled and simulated weight of the syndrome, incorrectly-guessed error bit distribution at the end of the first iteration, and two-itcrypteration Decoding Failure Rates (DFR)
  arXiv  Detail & Related papers  (2024-01-30T11:40:24Z)
• WR-ONE2SET: Towards Well-Calibrated Keyphrase Generation [57.11538133231843]
  Keyphrase generation aims to automatically generate short phrases summarizing an input document. The recently emerged ONE2SET paradigm generates keyphrases as a set and has achieved competitive performance. We propose WR-ONE2SET which extends ONE2SET with an adaptive instance-level cost Weighting strategy and a target Re-assignment mechanism.
  arXiv  Detail & Related papers  (2022-11-13T09:56:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.