SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting

• URL: http://arxiv.org/abs/2605.07243v1
• Date: Fri, 08 May 2026 04:59:48 GMT
• Title: SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting
• Abstract summary: SpecBlock is a block-iterative drafter that combines path dependence with cheap drafting.<n>A cost-aware bandit at deployment uses free verifier feedback to update the drafter selectively.
• Score: 25.273024580844346
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Speculative decoding accelerates LLM inference by drafting a tree of candidate continuations and verifying it in one target forward. Existing drafters fall into two camps with opposite weaknesses. Autoregressive drafters such as EAGLE-3 preserve dependence along each draft path but call the drafter once per tree depth, making drafting a non-trivial share of per-iteration latency. Parallel drafters cut drafter calls by predicting multiple future positions in one forward, but each position is predicted without seeing the others, producing paths the verifier rejects. In this paper, we propose SpecBlock, a block-iterative drafter that combines path dependence with cheap drafting. Each drafter forward produces K dependent positions and we call this a block. The draft tree grows through repeated block expansions. Two mechanisms explicitly carry path dependence to keep later draft positions accurate. Within each block, a layer-wise shift carries the previous position's hidden state into every decoder layer. Across blocks, each new block can start from any position of the previous block, inheriting its hidden state to extend the path. To spend verifier budget where acceptance is likely, a co-trained rank head replaces the fixed top-k tree by allocating per-position branching during drafting. To avoid training the drafter on prefixes it never produces at inference, a valid-prefix mask drops the loss at later positions once an earlier one is wrong. Beyond static drafting, a cost-aware bandit at deployment uses free verifier feedback to update the drafter selectively, only when the expected throughput gain exceeds the update cost. Experiments show that SpecBlock improves mean speedup by 8-13% over EAGLE-3 at 44-52% of its drafting cost, and cost-aware adaptation extends this lead to 11-19%.

Related papers

• Ceiling-Clipped Acceptance Histograms Indicate Stranded Speed-up in Block-Diffusion Speculative Decoding [0.008640713300591884]
  Speculative decoding speeds up generation with an efficient draft model (drafter)<n>High-acceptance block-diffusion drafters such as DFlash and DFlare fill an entire block in one parallel pass.<n>We post-train the drafter on a longer block with a short curriculum that emphasizes the newly exposed positions.
  arXiv  Detail & Related papers  (2026-08-31T08:23:14Z)
• Verification-Aware Training for Speculative Decoding [57.84976863792784]
  We introduce Verification-Aware Training (VAT), a plug-in framework that simulates verification at every training step and turns the resulting accept and reject patterns into supervision.<n>VAT modifies only the training objective, so it can be layered on top of existing methods without changing the draft architecture, the target model, or the inference procedure.<n>VAT improves average acceptance length by up to 11.4% and wall-clock speedup by up to 8.7%, with consistent gains across math, code, and chat benchmarks.
  arXiv  Detail & Related papers  (2026-08-31T01:42:10Z)
• DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees [40.61855455828842]
  Speculative decoding losslessly accelerates autoregressive language models by verifying multiple draft tokens in parallel.<n>Dartree is a training-free speculative decoding method that extends a pretrained AR correction head from chains to trees.
  arXiv  Detail & Related papers  (2026-08-13T17:43:44Z)
• DBLAST: Dependent Block Drafting for Stochastic Speculative Decoding [26.72775347772003]
  We show that the accepted draft length degrades as the entropy of the target sampling distribution increases.<n>We propose a dependent block drafter based on a low-rank latent mixture over token positions, complemented by an acceptance-oriented training objective.
  arXiv  Detail & Related papers  (2026-08-05T22:49:54Z)
• PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding [98.71600061506206]
  We propose PRESTO, a principled framework that extends tree-based drafting to diffusion drafters.<n> PRESTO achieves up to an average of $1.5times$ end-to-end throughput speedup on the state-of-the-art dedicated diffusion drafter SD.
  arXiv  Detail & Related papers  (2026-06-20T14:32:14Z)
• TreeFlash: Parallel AR-Approximation for Faster Speculative Decoding [32.75269650141292]
  One-shot block drafters for speculative decoding generate the full draft in a single forward pass.<n>They predict each draft token conditioned only on the prefix context, with no dependence on previously drafted tokens.<n>This non-autoregressive conditioning causes the drafter's distribution to diverge from the verifier's true autoregressive distribution as draft depth grows.<n>We propose TreeFlash, which addresses this by incorporating a layer conditioned on the drafter's hidden state and the previous token to approximate an autoregressive distribution.
  arXiv  Detail & Related papers  (2026-06-02T16:00:18Z)
• Draft-OPD: On-Policy Distillation for Speculative Draft Models [49.23782868133977]
  Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel.<n>We propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions.
  arXiv  Detail & Related papers  (2026-05-28T04:30:22Z)
• D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting [59.204113363599994]
  Speculative decoding accelerates inference by having a small drafter propose tokens that a larger target model verifies in parallel.<n>Recent diffusion-based parallel drafters such as DFlash predict the full B-token block in one forward pass, enabling deeper drafters and longer accepted blocks.<n>We derive per-position training weights from a differentiable surrogate of expected accepted draft length, matching the weight of each position to its log-probability contribution.<n>Across six benchmarks, two Qwen3-4B draft depths, two decoding temperatures, and two additional target models, D-PACE consistently improves both wall-clock speedup and average
  arXiv  Detail & Related papers  (2026-05-12T06:27:57Z)
• Accelerating Speculative Decoding with Block Diffusion Draft Trees [20.28933257827737]
  Speculative decoding accelerates autoregressive language models by using a lightweight drafter to propose multiple future tokens.<n>DFlash shows that a block diffusion drafter can generate an entire draft block in a single forward pass.<n> Vanilla DFlash verifies only a single drafted trajectory per round, potentially limiting its acceptance length.<n>We introduce DDTree, a method that constructs a draft tree directly from the per-position distributions of a block diffusion drafter.
  arXiv  Detail & Related papers  (2026-04-14T17:23:14Z)
• PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length [21.738896310075678]
  Speculative decoding (SD) is a powerful technique for accelerating the inference process of large language models (LLMs)<n>We propose Pacer, a novel approach that dynamically controls draft length using a lightweight, trainable pre-verification layer.<n>Our results demonstrate that Pacer achieves up to 2.66x Speedup over autoregressive decoding and consistently outperforms standard speculative decoding.
  arXiv  Detail & Related papers  (2026-02-01T15:12:38Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• FastEagle: Cascaded Drafting for Accelerating Speculative Decoding [6.482154864678126]
  We present FastEagle, a non-autoregressive cascaded drafter that emits an entire draft in a single forward pass.<n>FastEagle delivers substantial wall-clock speedups over strong autoregressive drafters while maintaining competitive acceptance behavior.
  arXiv  Detail & Related papers  (2025-09-24T09:38:32Z)
• Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding [73.67253077506672]
  Large language models (LLMs) deliver impressive generation quality, but incur very high inference cost.<n>Early-exit based self-speculative decoding (EESD) has emerged to mitigate this cost.<n>We propose Pipeline-Parallel Self-Speculative Decoding (PPSD) that fully pipelines the draft and verification work.
  arXiv  Detail & Related papers  (2025-09-19T04:51:41Z)
• Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass [72.07642648108849]
  Superposed Decoding is a new decoding algorithm that generates $k$ drafts at the cost of one autoregressive inference pass. Superposed Decoding can be combined with other decoding strategies, resulting in universal coverage gains when scaling inference time compute.
  arXiv  Detail & Related papers  (2024-05-28T17:40:48Z)
• Exploring and Improving Drafts in Blockwise Parallel Decoding [37.295672367973886]
  Blockwise parallel decoding (BPD) was proposed by Stern et al. as a method to improve inference speed of language models. This paper contributes to the understanding and improvement of block drafts in two ways. Experiments demonstrate that refined block drafts yield a +5-21% increase in block efficiency.
  arXiv  Detail & Related papers  (2024-04-14T11:49:38Z)
• Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding [65.94521678103237]
  Speculative decoding is a widely used method that accelerates the generation process of large language models. We introduce Ouroboros, which can generate draft phrases to parallelize the drafting process. Ouroboros can achieve speedups of up to $2.8times$ over speculative decoding and $3.9times$ over vanilla decoding.
  arXiv  Detail & Related papers  (2024-02-21T11:31:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.