D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting

• URL: http://arxiv.org/abs/2605.18810v1
• Date: Tue, 12 May 2026 06:27:57 GMT
• Title: D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting
• Abstract summary: Speculative decoding accelerates inference by having a small drafter propose tokens that a larger target model verifies in parallel.<n>Recent diffusion-based parallel drafters such as DFlash predict the full B-token block in one forward pass, enabling deeper drafters and longer accepted blocks.<n>We derive per-position training weights from a differentiable surrogate of expected accepted draft length, matching the weight of each position to its log-probability contribution.<n>Across six benchmarks, two Qwen3-4B draft depths, two decoding temperatures, and two additional target models, D-PACE consistently improves both wall-clock speedup and average
• Score: 59.204113363599994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding accelerates LLM inference by having a small drafter propose tokens that a larger target model verifies in parallel. Recent diffusion-based parallel drafters such as DFlash predict the full B-token block in one forward pass, enabling deeper drafters and longer accepted blocks. However, existing multi-token drafter objectives often use fixed position-dependent weighting schedules, such as head-dependent weights or block-position decays, which do not adapt as the positions limiting acceptance change during training. To address this, we derive per-position training weights from a differentiable surrogate of expected accepted draft length, matching the weight of each position to its log-probability gradient contribution. The resulting loss, D-PACE (Dynamic Position-Aware Cross-Entropy), shifts training signal toward positions that currently limit acceptance as the drafter improves. Across six benchmarks, two Qwen3-4B draft depths, two decoding temperatures, and two additional target models, D-PACE consistently improves both wall-clock speedup and average emitted length, with 2.3\% measured training-time overhead and no changes to the drafter architecture or inference procedure.

Related papers

• Verification-Aware Training for Speculative Decoding [57.84976863792784]
  We introduce Verification-Aware Training (VAT), a plug-in framework that simulates verification at every training step and turns the resulting accept and reject patterns into supervision.<n>VAT modifies only the training objective, so it can be layered on top of existing methods without changing the draft architecture, the target model, or the inference procedure.<n>VAT improves average acceptance length by up to 11.4% and wall-clock speedup by up to 8.7%, with consistent gains across math, code, and chat benchmarks.
  arXiv  Detail & Related papers  (2026-08-31T01:42:10Z)
• DBLAST: Dependent Block Drafting for Stochastic Speculative Decoding [26.72775347772003]
  We show that the accepted draft length degrades as the entropy of the target sampling distribution increases.<n>We propose a dependent block drafter based on a low-rank latent mixture over token positions, complemented by an acceptance-oriented training objective.
  arXiv  Detail & Related papers  (2026-08-05T22:49:54Z)
• AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters [67.51886052453452]
  Speculative decoding has become a prevalent paradigm for accelerating large language model inference.<n>Recent work such as DFlash further boosts drafting efficiency by leveraging diffusion drafters.<n>We propose AdaFlash framework, comprising two components: (i) an on-policy distillation (OPD) algorithm with reverse-KL divergence tailored for diffusion drafters, and (ii) an adaptive length head that dynamically adjusts the candidate sequence length on the fly.
  arXiv  Detail & Related papers  (2026-07-21T15:52:50Z)
• Teaching Diffusion to Speculate Left-to-Right [3.205247598097648]
  Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs.<n> Speculative decoding addresses this bottleneck by employing a lightweight draft model to propose multiple future tokens that are subsequently verified in parallel by a larger target model.<n>Recent work has demonstrated that diffusion language models are well suited for this setting, as they can generate entire blocks of draft tokens in parallel.
  arXiv  Detail & Related papers  (2026-06-10T01:21:56Z)
• Draft-OPD: On-Policy Distillation for Speculative Draft Models [49.23782868133977]
  Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel.<n>We propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions.
  arXiv  Detail & Related papers  (2026-05-28T04:30:22Z)
• FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration [21.05266483397106]
  We introduce FlexDraft, a speculative decoding framework that adapts to varying batch sizes through three key designs.<n> Attention Tuning enables block diffusion drafting by tuning only the attention projectors of the final few layers on mask tokens.<n>Bonus-guided uses a lightweight conditioned on the resolved bonus token to calibrate draft logits.
  arXiv  Detail & Related papers  (2026-05-19T15:48:16Z)
• Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendation [27.749196490846916]
  PAD-Rec is a lightweight module that augments the draft model with two complementary signals.<n>Item position embeddings explicitly encode the within-item slot of each token.<n>Step position embeddings encode the draft step, allowing the model to adapt to depth-dependent uncertainty.
  arXiv  Detail & Related papers  (2026-04-30T11:37:08Z)
• SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration [13.454534256560558]
  Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models.<n>We propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision.<n>We show that our method achieves up to 2.33x wall-time speedup over standard autoregressive decoding.
  arXiv  Detail & Related papers  (2026-04-14T03:47:04Z)
• Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning [67.88087883391475]
  We introduce Learning to Draft, a novel method that directly optimize for throughput of each draft-and-verify cycle.<n> LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to 36.4%.
  arXiv  Detail & Related papers  (2026-03-02T09:17:48Z)
• PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length [21.738896310075678]
  Speculative decoding (SD) is a powerful technique for accelerating the inference process of large language models (LLMs)<n>We propose Pacer, a novel approach that dynamically controls draft length using a lightweight, trainable pre-verification layer.<n>Our results demonstrate that Pacer achieves up to 2.66x Speedup over autoregressive decoding and consistently outperforms standard speculative decoding.
  arXiv  Detail & Related papers  (2026-02-01T15:12:38Z)
• Beat the long tail: Distribution-Aware Speculative Decoding for RL Training [75.75462952580796]
  We propose a Distribution Aware Speculative decoding framework that accelerates RL rollouts without altering model outputs.<n>Experiments on math and code reasoning tasks show that DAS reduces rollout time up to 50% while preserving identical training curves.
  arXiv  Detail & Related papers  (2025-11-17T19:02:12Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• SeqPE: Transformer with Sequential Position Encoding [76.22159277300891]
  SeqPE represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings.<n> Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM) and accuracy--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign.
  arXiv  Detail & Related papers  (2025-06-16T09:16:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.