Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing

• URL: http://arxiv.org/abs/2605.14978v2
• Date: Fri, 15 May 2026 02:45:40 GMT
• Title: Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing
• Abstract summary: Speculative efficiency is often bottlenecked by hard-to-draft positions.<n>Most learning-based drafters are still optimized with token-level supervised objectives.<n>We propose PPOW, a reinforcement learning framework that shifts drafter optimization from token-level imitation to window-level optimization.
• Score: 14.853472940422199
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speculative decoding accelerates LLM inference by having a lightweight draft model propose speculative windows of candidate tokens for parallel verification by a larger target model. In practice, speculative efficiency is often bottlenecked by hard-to-draft positions, where an early mismatch truncates the accepted prefix and invalidates the rest of the speculative window. Most learning-based drafters are still optimized with token-level supervised objectives, even though speculative utility is inherently window-level and prefix-sensitive. We propose PPOW (Performance-Driven Policy Optimization with Adaptive Windowing), a reinforcement learning framework that shifts drafter optimization from token-level imitation to window-level optimization. PPOW combines a Cost-Aware Speedup Reward, a Distribution-Based Proximity Reward, and Adaptive Divergence-Aware Windowing, which prioritizes informative windows with high confidence-weighted draft-target divergence. PPOW achieves average acceptance lengths of 6.29-6.52 and speedups of 3.39-4.36$\times$ across multiple model families and benchmarks under a unified decoding protocol. These results show that performance-driven window-level optimization is a practical approach to improving speculative decoding efficiency.

Related papers

• Effect of Abstractions and Prompting Strategies on LLM-Guided High-Performance Optimizations [0.0]
  Large Language Models (LLMs) are capable of generating and understanding code.<n>Traditionally, this automation involves transcribing the source code into a domain-specific representation.<n>This paper investigates whether the traditional abstractions used in automated code optimization improve the performance and correctness of LLM-guided optimizations.
  arXiv  Detail & Related papers  (2026-08-08T12:12:28Z)
• Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders [74.55181072260713]
  Training with beam-search negatives consistently outperforms random negatives.<n>We introduce Windowed Partial AUC (WPAUC), which constrains the false positive rate (FPR) to a window to more directly align with Top-$K$ metrics.<n>Experiments on four real-world datasets validate the theory and deliver consistent state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-24T12:31:57Z)
• Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs [18.579725648410815]
  We propose Adaptive Prompt Structure Factorization (aPSF), an API-only framework (prompt-in/text-out; no access to model internals)<n>aPSF uses an Architect model to discover task-specific prompt structures as semantic factors.<n>aPSF outperforms strong baselines including principle-awares, improving accuracy by up to +2.16 percentage points on average.
  arXiv  Detail & Related papers  (2026-04-08T05:25:40Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• PerfCoder: Large Language Models for Interpretable Code Performance Optimization [15.79612555952707]
  PerfCoder is a family of large language models (LLMs) designed to generate performance-enhanced code from source code.<n>PerfCoder is fine-tuned on a curated collection of real-world optimization trajectories with human-readable annotations.<n>PerfCoder surpasses all existing models in both runtime speedup and effective optimization rate.
  arXiv  Detail & Related papers  (2025-12-16T02:30:04Z)
• HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs [54.16300997612526]
  Large Language Models (LLMs) increasingly rely on Chain-of-Thought (CoT) reasoning to improve accuracy on complex tasks.<n>This paper introduces the Hybrid Policy Optimization (i.e., HiPO), a framework for adaptive reasoning control.<n> Experiments across mathematics and coding benchmarks demonstrate that HiPO can substantially reduce token length while maintaining or improving accuracy.
  arXiv  Detail & Related papers  (2025-09-28T16:46:12Z)
• Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs [9.085280547983091]
  DeBoP is an automatic optimization method, which focuses on the optimization directly on the behavior of LwLLMs.<n>We evaluate DeBoP on seven challenging tasks where state-of-the-art LLMs excel but LwLLMs generally underperform.<n>DeBoP-optimized LwLLMs surpass GPT-3.5 on most tasks while reducing computational time by approximately 60%.
  arXiv  Detail & Related papers  (2025-06-06T02:40:42Z)
• ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning [14.034412856423529]
  Direct Preference Optimization (DPO) has gained attention for its simplicity and computational efficiency in aligning large language models (LLMs)<n>Recent advancements have extended DPO to multimodal scenarios, achieving strong performance.<n>Traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness.<n>We propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization.
  arXiv  Detail & Related papers  (2025-05-25T11:33:08Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Towards Efficient Exact Optimization of Language Model Alignment [93.39181634597877]
  Direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. We show that DPO derived based on the optimal solution of problem leads to a compromised mean-seeking approximation of the optimal solution in practice. We propose efficient exact optimization (EXO) of the alignment objective.
  arXiv  Detail & Related papers  (2024-02-01T18:51:54Z)
• Robust Prompt Optimization for Large Language Models Against Distribution Shifts [80.6757997074956]
  Large Language Model (LLM) has demonstrated significant ability in various Natural Language Processing tasks. We propose a new problem of robust prompt optimization for LLMs against distribution shifts. This problem requires the prompt optimized over the labeled source group can simultaneously generalize to an unlabeled target group.
  arXiv  Detail & Related papers  (2023-05-23T11:30:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.