Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding

• URL: http://arxiv.org/abs/2509.22134v1
• Date: Fri, 26 Sep 2025 09:55:35 GMT
• Title: Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding
• Abstract summary: We introduce Group Tree Optimization (GTO), which aligns training with the decoding-time tree policy.<n>We prove that increasing Draft Tree Reward provably improves acceptance length and speedup.<n>GTO offers a practical, general solution for efficient large language model inference.
• Score: 24.681973968208364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speculative decoding accelerates large language model (LLM) inference by letting a lightweight draft model propose multiple tokens that the target model verifies in parallel. Yet existing training objectives optimize only a single greedy draft path, while decoding follows a tree policy that re-ranks and verifies multiple branches. This draft policy misalignment limits achievable speedups. We introduce Group Tree Optimization (GTO), which aligns training with the decoding-time tree policy through two components: (i) Draft Tree Reward, a sampling-free objective equal to the expected acceptance length of the draft tree under the target model, directly measuring decoding performance; (ii) Group-based Draft Policy Training, a stable optimization scheme that contrasts trees from the current and a frozen reference draft model, forming debiased group-standardized advantages and applying a PPO-style surrogate along the longest accepted sequence for robust updates. We further prove that increasing our Draft Tree Reward provably improves acceptance length and speedup. Across dialogue (MT-Bench), code (HumanEval), and math (GSM8K), and multiple LLMs (e.g., LLaMA-3.1-8B, LLaMA-3.3-70B, Vicuna-1.3-13B, DeepSeek-R1-Distill-LLaMA-8B), GTO increases acceptance length by 7.4% and yields an additional 7.7% speedup over prior state-of-the-art EAGLE-3. By bridging draft policy misalignment, GTO offers a practical, general solution for efficient LLM inference.

Related papers

• DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees [40.61855455828842]
  Speculative decoding losslessly accelerates autoregressive language models by verifying multiple draft tokens in parallel.<n>Dartree is a training-free speculative decoding method that extends a pretrained AR correction head from chains to trees.
  arXiv  Detail & Related papers  (2026-08-13T17:43:44Z)
• PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding [98.71600061506206]
  We propose PRESTO, a principled framework that extends tree-based drafting to diffusion drafters.<n> PRESTO achieves up to an average of $1.5times$ end-to-end throughput speedup on the state-of-the-art dedicated diffusion drafter SD.
  arXiv  Detail & Related papers  (2026-06-20T14:32:14Z)
• Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting [36.83545196908392]
  BASTION is a budget-aware speculative decoding framework with tree-based diffusion drafting.<n>Our framework integrates three synergistic components: (1) an acceptance surrogate that estimates expected accepted length via path confidence, (2) an online latency estimator that calibrates a hardware-aware roofline model, and (3) an adaptive best-first expansion that grows the tree until marginal gains no longer justify incremental verification costs.
  arXiv  Detail & Related papers  (2026-05-28T10:21:34Z)
• UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding [2.486699239459455]
  We propose a unified perspective that casts tree-based verification as a conditional OT problem.<n>We introduce UniVer, a verification algorithm that jointly optimize across tree levels by composing local optimal transport plans.
  arXiv  Detail & Related papers  (2026-05-06T06:42:58Z)
• SMART: When is it Actually Worth Expanding a Speculative Tree? [26.986851887229054]
  Tree-based speculative decoding accelerates autoregressive generation by verifying a branching tree draft tokens in a single targetmodel forward pass.<n>Existing methods prioritize maximizing token-level likelihood of the number of accepted tokens while ignoring a critical efficiency paradox''<n>We propose SMART, a system-aware marginal analysis framework for runtime tree construction.
  arXiv  Detail & Related papers  (2026-04-09T13:17:56Z)
• LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding [67.61563011564388]
  Speculative decoding accelerates autoregressive large language model (LLM) inference.<n>Standard training minimizes Kullback-Leibler (KL) divergence as a proxy objective.<n>We propose LK losses, special training objectives that directly target acceptance rate.
  arXiv  Detail & Related papers  (2026-02-27T10:20:11Z)
• Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding [35.984745508100595]
  We present a systematic evaluation of verification strategies across model families, tasks, and sampling regimes.<n>Traversal Verification dominates consistently, with OT-based methods lagging far behind.<n>We propose delayed tree expansion, which drafts a partial single path, delaying the i.i.d. branching point.
  arXiv  Detail & Related papers  (2026-02-19T01:41:58Z)
• iGRPO: Self-Feedback-Driven LLM Reasoning [88.83313431248473]
  Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions.<n>We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts.<n>Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models.
  arXiv  Detail & Related papers  (2026-02-09T18:45:11Z)
• SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decoding [15.734450444255787]
  Speculative decoding has emerged as a promising approach to accelerate inference in vision-language models.<n>Existing methods rely on static tree structures that remain fixed throughout the decoding process.<n>We propose SAGE, a novel framework that dynamically adjusts the speculation tree structure based on real-time prediction uncertainty.
  arXiv  Detail & Related papers  (2026-01-31T05:35:40Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• FastEagle: Cascaded Drafting for Accelerating Speculative Decoding [6.482154864678126]
  We present FastEagle, a non-autoregressive cascaded drafter that emits an entire draft in a single forward pass.<n>FastEagle delivers substantial wall-clock speedups over strong autoregressive drafters while maintaining competitive acceptance behavior.
  arXiv  Detail & Related papers  (2025-09-24T09:38:32Z)
• TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling [65.46347858249295]
  TreePO is a self-guided rollout algorithm that views sequence generation as a tree-structured searching process.<n>TreePO essentially reduces the per-update compute burden while preserving or enhancing exploration diversity.
  arXiv  Detail & Related papers  (2025-08-24T16:52:37Z)
• TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree [52.44403214958304]
  In this paper, we introduce TreeLoRA, a novel approach that constructs layer-wise adapters by leveraging hierarchical gradient similarity.<n>To reduce the computational burden of task similarity estimation, we employ bandit techniques to develop an algorithm based on lower confidence bounds.<n> experiments on both vision transformers (ViTs) and large language models (LLMs) demonstrate the effectiveness and efficiency of our approach.
  arXiv  Detail & Related papers  (2025-06-12T05:25:35Z)
• RASD: Retrieval-Augmented Speculative Decoding [5.3926068062773895]
  Speculative decoding accelerates inference in large language models (LLMs)<n>This paper proposes RASD (Retrieval-Augmented Speculative Decoding), which adopts retrieval methods to enhance model-based speculative decoding.
  arXiv  Detail & Related papers  (2025-03-05T12:10:14Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure [40.9990864658776]
  Speculative decoding employs a "draft and then verify" mechanism to allow multiple tokens to be generated in one step.<n>Existing methods mainly adopt fixed draft structures, which fail to adapt to different situations.<n>We propose OPT-Tree, an algorithm to construct adaptive and scalable draft trees.
  arXiv  Detail & Related papers  (2024-06-25T04:45:53Z)
• Tree Prompting: Efficient Task Adaptation without Fine-Tuning [112.71020326388029]
  Tree Prompting builds a decision tree of prompts, linking multiple LM calls together to solve a task. Experiments on classification datasets show that Tree Prompting improves accuracy over competing methods and is competitive with fine-tuning.
  arXiv  Detail & Related papers  (2023-10-21T15:18:22Z)
• Tree-Planner: Efficient Close-loop Task Planning with Large Language Models [63.06270302774049]
  Tree-Planner reframes task planning with Large Language Models into three distinct phases. Tree-Planner achieves state-of-the-art performance while maintaining high efficiency.
  arXiv  Detail & Related papers  (2023-10-12T17:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.