FASTER: Value-Guided Sampling for Fast RL

• URL: http://arxiv.org/abs/2604.19730v1
• Date: Tue, 21 Apr 2026 17:52:17 GMT
• Title: FASTER: Value-Guided Sampling for Fast RL
• Abstract summary: FASTER is a method for getting the benefits of sampling-based test-time scaling of diffusion-based policies without the computational cost.<n>FASTER achieves the same performance while substantially reducing training and inference compute requirements.
• Score: 103.55398181003262
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Some of the most performant reinforcement learning algorithms today can be prohibitively expensive as they use test-time scaling methods such as sampling multiple action candidates and selecting the best one. In this work, we propose FASTER, a method for getting the benefits of sampling-based test-time scaling of diffusion-based policies without the computational cost by tracing the performance gain of action samples back to earlier in the denoising process. Our key insight is that we can model the denoising of multiple action candidates and selecting the best one as a Markov Decision Process (MDP) where the goal is to progressively filter action candidates before denoising is complete. With this MDP, we can learn a policy and value function in the denoising space that predicts the downstream value of action candidates in the denoising process and filters them while maximizing returns. The result is a method that is lightweight and can be plugged into existing generative RL algorithms. Across challenging long-horizon manipulation tasks in online and batch-online RL, FASTER consistently improves the underlying policies and achieves the best overall performance among the compared methods. Applied to a pretrained VLA, FASTER achieves the same performance while substantially reducing training and inference compute requirements. Code is available at https://github.com/alexanderswerdlow/faster .

Related papers

• Flash-BoN: Instant Drafts for Inference-Time Scaling in Diffusion Models [91.19049123903137]
  Inference-time scaling for text-to-image generation has progressed from simple Best-of-$N$ sampling to guided search methods.<n>These approaches focus on when and how often to verify during denoising but largely treat the cost of generation itself as fixed.<n>We show that under wall-clock evaluation, simple BoN already matches or outperforms several guided search techniques.
  arXiv  Detail & Related papers  (2026-07-05T19:02:32Z)
• QPILOTS: Efficient Test-Time Q-Steering for Flow Policies [20.217020870532686]
  QPILOTS is a method that leaves the original policy unmodified and steers the denoising process at inference time.<n>On a standard offline-to-online RL benchmark, QPILOTS achieves the best aggregate performance, reaching an average success rate of 90% across 50 tasks.
  arXiv  Detail & Related papers  (2026-06-11T18:22:03Z)
• $S^3$: Stratified Scaling Search for Test-Time in Diffusion Language Models [10.157422365382933]
  Test-time scaling investigates whether a fixed diffusion language model (DLM) can generate better outputs when given more inference compute.<n>We propose $S3$ (Stratified Scaling Search), a verifier-guided search method that improves generation by reallocating compute during the denoising process.<n>Experiments with LLaDA-8B-Instruct on MATH-500, GSM8K, ARC-Challenge, and TruthfulQA demonstrate that $S3$ consistently improves performance across benchmarks.
  arXiv  Detail & Related papers  (2026-04-07T00:51:06Z)
• TTSnap: Test-Time Scaling of Diffusion Models via Noise-Aware Pruning [53.52543819839442]
  A prominent approach to test-time scaling for text-to-image diffusion models formulates the problem as a search over multiple noise seeds.<n>We propose test-time scaling with noise-aware pruning (TTSnap), a framework that prunes low-quality candidates without fully denoising them.
  arXiv  Detail & Related papers  (2025-11-27T09:14:26Z)
• Real-Time Iteration Scheme for Diffusion Policy [23.124189676943757]
  We introduce a novel approach inspired by the Real-Time Iteration (RTI) Scheme to accelerate inference.<n>We propose a scaling-based method to effectively handle discrete actions, such as grasping, in robotic manipulation.<n>The proposed scheme significantly reduces runtime computational costs without the need for distillation or policy redesign.
  arXiv  Detail & Related papers  (2025-08-07T13:49:00Z)
• From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization [7.531052649961168]
  Reinforcement learning with verifiable rewards (RLVR) has recently advanced the reasoning capabilities of large language models (LLMs)<n>We investigate RLVR from a sample-centric perspective and introduce LPPO, a framework of progressive optimization techniques.<n>Our work addresses a critical question: how to best leverage a small set of trusted, high-quality demonstrations, rather than simply scaling up data volume.
  arXiv  Detail & Related papers  (2025-07-09T06:05:28Z)
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [65.18157595903124]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiences [8.983448736644382]
  Efficient utilization of the replay buffer plays a significant role in the off-policy actor-critic reinforcement learning (RL) algorithms. We propose a method for achieving sample efficiency, which focuses on selecting unique samples and adding them to the replay buffer.
  arXiv  Detail & Related papers  (2024-02-05T10:04:00Z)
• DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion [137.8749239614528]
  We propose a new formulation of temporal action detection (TAD) with denoising diffusion, DiffTAD. Taking as input random temporal proposals, it can yield action proposals accurately given an untrimmed long video.
  arXiv  Detail & Related papers  (2023-03-27T00:40:52Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.