Speculative Sampling with Reinforcement Learning

• URL: http://arxiv.org/abs/2601.12212v1
• Date: Sun, 18 Jan 2026 01:31:29 GMT
• Title: Speculative Sampling with Reinforcement Learning
• Authors: Chenan Wang, Daniel H. Shi, Haipeng Chen,
• Abstract summary: Inference time latency has remained an open challenge for real world applications of large language models (LLMs)<n>We introduce Reinforcement learning for Speculative Sampling (Re-SpS), the first reinforcement learning (RL)-based framework for draft tree hyper parameter optimization.
• Score: 6.146330145308428
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inference time latency has remained an open challenge for real world applications of large language models (LLMs). State-of-the-art (SOTA) speculative sampling (SpS) methods for LLMs, like EAGLE-3, use tree-based drafting to explore multiple candidate continuations in parallel. However, the hyperparameters controlling the tree structure are static, which limits flexibility and efficiency across diverse contexts and domains. We introduce Reinforcement learning for Speculative Sampling (Re-SpS), the first reinforcement learning (RL)-based framework for draft tree hyperparameter optimization. Re-SpS dynamically adjusts draft tree hyperparameters in real-time, learning context-aware policies that maximize generation speed by balancing speculative aggression with computational overhead. It leverages efficient state representations from target model hidden states and introduces multi-step action persistence for better context modeling. Evaluation results across five diverse benchmarks demonstrate consistent improvements over the SOTA method EAGLE-3, achieving up to 5.45$\times$ speedup over the backbone LLM and up to 1.12$\times$ speedup compared to EAGLE-3 across five diverse benchmarks, with no loss in output fidelity.

Related papers

• AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineering [52.67783579040657]
  AceGRPO is a machine learning system that prioritizes tasks at the agent's learning frontier to maximize learning efficiency.<n>Our trained Ace-30B model achieves a 100% valid submission rate on MLE-Bench-Lite, approaches the performance of proprietary frontier models, and outperforms larger open-source baselines.
  arXiv  Detail & Related papers  (2026-02-08T10:55:03Z)
• TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs [67.55973229034319]
  This paper introduces TempSamp-R1, a new reinforcement fine-tuning framework designed to improve the effectiveness of adapting multimodal large language models (MLLMs) to video temporal grounding tasks.<n>We show that TempSamp-R1 outperforms GRPO-based baselines, establishing new state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-22T17:30:15Z)
• Agentic Reinforced Policy Optimization [66.96989268893932]
  Large-scale reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks.<n>Current RL algorithms inadequately balance the models' intrinsic long-horizon reasoning capabilities and their proficiency in multi-turn tool interactions.<n>We propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents.
  arXiv  Detail & Related papers  (2025-07-26T07:53:11Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• Large Language Model Enhanced Particle Swarm Optimization for Hyperparameter Tuning for Deep Learning Models [2.3949320404005436]
  Particle Swarm Optimization and Large Language Models (LLMs) have been individually applied in optimization and deep learning.<n>Our work addresses this gap by integrating LLMs into PSO to reduce model evaluations and improve convergence.<n>Our method speeds up search space exploration by substituting underperforming particle placements with best suggestions.
  arXiv  Detail & Related papers  (2025-04-19T00:54:59Z)
• Prism: Dynamic and Flexible Benchmarking of LLMs Code Generation with Monte Carlo Tree Search [13.135962181354465]
  Static benchmarks fail to capture the depth and breadth of Large Language Models (LLMs) capabilities.<n>We introduce Prism, a flexible, dynamic benchmarking framework designed for comprehensive LLM assessment.<n> Prism builds on three key components: (1) a tree-based state representation that models evaluation as a Markov Decision Process, (2) a Monte Carlo Tree Search algorithm adapted to uncover challenging evaluation scenarios, and (3) a multi-agent evaluation pipeline that enables simultaneous assessment of diverse capabilities.
  arXiv  Detail & Related papers  (2025-04-07T20:53:18Z)
• Small but Mighty: Enhancing Time Series Forecasting with Lightweight LLMs [11.1036247482657]
  We present SMETimes, the first systematic investigation of sub-3B parameter SLMs for efficient and accurate time series forecasting.<n>Our approach centers on three key innovations: A statistically-enhanced prompting mechanism that bridges numerical time series with textual semantics through statistical features; A adaptive fusion embedding architecture that aligns temporal patterns with language model token spaces through learnable parameters.
  arXiv  Detail & Related papers  (2025-03-05T15:27:36Z)
• From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning [31.95005389919542]
  Scaling data and model size has been proven effective for boosting the performance of large language models.<n>In this work, we introduce Aggregation Fine-Tuning (AFT), a supervised finetuning paradigm.<n> Empirical evaluations on benchmark datasets show that AFT-trained models substantially outperform standard SFT.
  arXiv  Detail & Related papers  (2025-01-21T04:11:59Z)
• SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models [88.29990536278167]
  We introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs.<n>Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
  arXiv  Detail & Related papers  (2024-12-16T09:47:43Z)
• Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMs [29.735465300269993]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks, yet they often struggle with spatial reasoning.<n>This paper presents a novel neural-symbolic framework that enhances LLMs' spatial reasoning abilities through iterative feedback between LLMs and Answer Set Programming (ASP)<n>We evaluate our approach on two benchmark datasets: StepGame and SparQA.
  arXiv  Detail & Related papers  (2024-11-27T18:04:05Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning that learns the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model.<n>We achieve this by learning an underlying Bernoulli distribution to sample binary pruning masks.<n>Experiments conducted on LLaMA, LLaMA-2, LLaMA-3, Vicuna, and Mistral models demonstrate the promising performance of our method in efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.