Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models

• URL: http://arxiv.org/abs/2607.16506v1
• Date: Fri, 17 Jul 2026 21:00:28 GMT
• Title: Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models
• Abstract summary: Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly.<n>We show this failure mode in residual reinforcement learning over a frozen VLA base policy.<n>We propose Residual RL, which optimize handoff quality by augmenting each subtask's sparse success reward with an offline-estimated foresight value.
• Score: 32.14222486099161
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly due to long-horizon credit assignment and subtask coupling: a state that is geometrically successful for the current skill can be brittle for downstream skills. We show this failure mode in residual reinforcement learning (RL) over a frozen VLA base policy: constant sparse success rewards improve each subtask in isolation yet yield little or no gain when skills are chained, because terminal state quality is uncontrolled. We propose Foresight Residual RL, which optimizes handoff quality by augmenting each subtask's sparse success reward with an offline-estimated foresight value -- the probability of future subtask success conditioned on the terminal state of the current subtask. Concretely, we (i) train a visual foresight predictor from images of terminal states of the base policy, labeled using downstream rollout statistics, and (ii) train residual policies via backward foresight induction, using the predictor output as a reward multiplier. On a three-phase wrench-based nut-tightening assembly task in Isaac Gym (grasp, move-insert, rotate), our method achieves 85.6% full-task success, outperforming standard subtask residual RL (54.5%) and VLA baselines, while leaving per-subtask success unchanged. These results highlight that improving long-horizon performance requires shaping which successful states are produced at each sub-task, not only whether success occurs.

Related papers

• Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals [1.7496207597922033]
  Contrastive reinforcement learning scales effectively in goal-conditioned tasks by casting policy learning into a self-supervised contrastive objective.<n>Our theoretical analysis shows that this omission induces a systematic overestimation bias in goal-reaching values.<n>We introduce two minimal yet strong corrections: mass-weighted InfoNCE corrects the overweighting of short surviving futures in critic learning, and a log-survival-mass score restores the missing survival mass in policy optimization.
  arXiv  Detail & Related papers  (2026-08-27T03:24:56Z)
• When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents [62.10623192614306]
  Extracting skills from past successes is critical for the efficient evolution of Large Language Model (LLM) agents.<n>We propose textbfBoundary-Aware Skill Memory (BASM), which augments each skill with explicit boundary fields.<n>BASM consistently outperforms success-distilled skill-memory baselines.
  arXiv  Detail & Related papers  (2026-08-23T10:16:06Z)
• RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models [8.012549967082832]
  $RL2$ is an adaptive inference-time steering framework for Vision-Language-Action (VLA) models.<n>We train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert.<n>This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL.
  arXiv  Detail & Related papers  (2026-07-29T14:49:56Z)
• RL Forgets! Towards Continual Policy Optimization [57.4336338996653]
  Continual post-training is becoming a central paradigm for adapting vision-language models to evolving tasks.<n>Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting.<n>We propose Continual Policy Optimization (CPO), a replay-free framework grounded in a prior-task behavioral KL objective.
  arXiv  Detail & Related papers  (2026-07-05T15:46:11Z)
• Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models [60.17417174639865]
  Vision-Language-Action (VLA) models acquire broad embodied capabilities through large-scale pretraining.<n>The prevailing remedy, post-training, improves task-specific performance but narrows the generalist capability that makes pretraining valuable.<n>We propose SVA, a framework that equips frozen VLA policies with long-term consequence awareness.
  arXiv  Detail & Related papers  (2026-07-04T07:53:10Z)
• Learning Process Rewards via Success Visitation Matching for Efficient RL [54.67547527674048]
  We propose a simple approach to transform a sparse outcome reward into a dense process reward.<n>Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes.<n>By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion.
  arXiv  Detail & Related papers  (2026-06-22T17:30:24Z)
• Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes [40.20394793623452]
  When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome.<n>We propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for viability and efficiency.<n>HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.
  arXiv  Detail & Related papers  (2026-06-15T17:57:14Z)
• EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models [84.73890225707264]
  We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies.<n>Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data.<n>We release an open-source robustness with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.
  arXiv  Detail & Related papers  (2026-05-25T06:31:03Z)
• Contrastive Conceptor Activation Steering (COAST): Unlocking Vision-Language-Action Models through Hidden States [4.1864205728857256]
  Vision-Language-Action (VLA) models leverage powerful perceptual priors from web-scale Vision-Language Model (VLM) pre-training.<n>To mitigate this, we propose Contrastive Conceptor Activation Steering (COAST)<n>COAST uses conceptors to identify success-critical subspaces for a target robotic task.
  arXiv  Detail & Related papers  (2026-05-16T20:28:21Z)
• RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models [90.39703013636868]
  RePO-VLA is a recovery-driven policy optimization framework.<n>It assigns distinct roles to success, recovery, and failure trajectories.<n>It raises adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
  arXiv  Detail & Related papers  (2026-05-10T08:24:05Z)
• Generalizable Dense Reward for Long-Horizon Robotic Tasks [10.301325532021218]
  Vision-Language Models (VLMs) decompose tasks into verifiable subtasks.<n>Self-certainty provides per-step intrinsic guidance throughout PPO finetuning.<n>On the CHORES benchmark,VLLR achieves up to 56% absolute success rate gains over the pretrained policy.
  arXiv  Detail & Related papers  (2026-03-31T02:05:07Z)
• PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation [27.791908160098625]
  PALM builds policy learning around interaction-centric affordance reasoning and subtask progress cues.<n>Palm consistently outperforms baselines in simulation and real-world experiments.
  arXiv  Detail & Related papers  (2026-01-11T21:00:58Z)
• Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic [42.57662196581823]
  Learning high-quality $Q$-value functions plays a key role in the success of many modern off-policy deep reinforcement learning (RL) algorithms. Deviating from the common viewpoint, we observe that $Q$-values are often underestimated in the latter stage of the RL training process. We propose the Blended Exploitation and Exploration (BEE) operator, a simple yet effective approach that updates $Q$-value using both historical best-performing actions and the current policy.
  arXiv  Detail & Related papers  (2023-06-05T13:38:14Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.