Learning More from Less: Reinforcement Learning from Hindsight

• URL: http://arxiv.org/abs/2607.09042v1
• Date: Fri, 10 Jul 2026 02:17:41 GMT
• Title: Learning More from Less: Reinforcement Learning from Hindsight
• Abstract summary: Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models.<n>We present Learning from Hindsight (LfH), which brings hindsight relabeling to RL post-training ofVLAs by scoring failed rollouts against the tasks they actually achieved.
• Score: 19.802783289526598
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern. Manipulation tasks typically provide only sparse rewards, so a weak policy fails almost every rollout early in training and has little to learn from, even when those failures execute coherent behavior. Such a failure, however, is a success at a different task. We present Learning from Hindsight (LfH), which brings hindsight relabeling to RL post-training of VLAs by scoring failed rollouts against the tasks they actually achieved. A single vision-language model relabels both the instruction and the reward, proposing a hindsight instruction for a group of failed rollouts and scoring how well each satisfies it, and the policy trains on the relabeled and original rollouts jointly. Because VLAs generalize across language, relabeling in language lets the policy learn more from the same trajectories. On out-of-distribution LIBERO-PRO tasks, where standard RL improves only slowly, LfH achieves $5\times$ improvement in sample efficiency, and outperforms a dense progress-reward baseline. The gains hold across VLA backbones and on a physical Franka robot.

Related papers

• RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models [8.012549967082832]
  $RL2$ is an adaptive inference-time steering framework for Vision-Language-Action (VLA) models.<n>We train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert.<n>This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL.
  arXiv  Detail & Related papers  (2026-07-29T14:49:56Z)
• EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models [84.73890225707264]
  We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies.<n>Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data.<n>We release an open-source robustness with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.
  arXiv  Detail & Related papers  (2026-05-25T06:31:03Z)
• RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models [90.39703013636868]
  RePO-VLA is a recovery-driven policy optimization framework.<n>It assigns distinct roles to success, recovery, and failure trajectories.<n>It raises adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
  arXiv  Detail & Related papers  (2026-05-10T08:24:05Z)
• Jump-Start Reinforcement Learning with Vision-Language-Action Regularization [1.2599533416395767]
  Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation.<n>Current limitations hinder their direct use in fast and precise manipulation.<n>We propose Vision-Language-Action Jump-Starting (VLAJS) to improve exploration and learning efficiency.
  arXiv  Detail & Related papers  (2026-04-15T11:17:54Z)
• Generalizable Dense Reward for Long-Horizon Robotic Tasks [10.301325532021218]
  Vision-Language Models (VLMs) decompose tasks into verifiable subtasks.<n>Self-certainty provides per-step intrinsic guidance throughout PPO finetuning.<n>On the CHORES benchmark,VLLR achieves up to 56% absolute success rate gains over the pretrained policy.
  arXiv  Detail & Related papers  (2026-03-31T02:05:07Z)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving [75.29519604607111]
  Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs)<n>We show that when the initial direction or quality of the chain-of-thought (CoT) is suboptimal, the model often fails to reach the correct answer.<n>We introduce Reinforcement Learning with Re-solving (Re$2$), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary.
  arXiv  Detail & Related papers  (2026-03-07T13:17:46Z)
• Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning [30.941068379610794]
  We study the behavior of large-scale pretrained Vision-Language-Action (VLA) models trained from scratch.<n>We find that pretrained VLAs are remarkably resistant to forgetting compared with smaller policy models trained from scratch.<n>Our analysis reveals that pretraining plays a critical role in downstream continual learning performance.
  arXiv  Detail & Related papers  (2026-03-04T08:03:13Z)
• Learning Affordances at Inference-Time for Vision-Language-Action Models [50.93181349331096]
  In robotics, Vision-Language-Action models (VLAs) offer a promising path towards solving complex control tasks.<n>We introduce Learning from Inference-Time Execution (LITEN), which connects a VLA low-level policy to a high-level VLM that conditions on past experiences.<n>Our approach iterates between a reasoning phase that generates and executes plans for the low-level VLA, and an assessment phase that reflects on the resulting execution.
  arXiv  Detail & Related papers  (2025-10-22T16:43:29Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning [81.7764584515496]
  Vision-Language-Action (VLA) models have emerged as a powerful paradigm for robotic manipulation.<n>These models face two fundamental challenges: scarcity and high cost of large-scale human-operated robotic trajectories.<n>We introduce SimpleVLA-RL, an efficient reinforcement learning framework tailored for VLA models.
  arXiv  Detail & Related papers  (2025-09-11T17:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.