Phase-and-First-Arrival VLM Feedback for Sparse-Reward Reinforcement Learning in Surgical Manipulation

• URL: http://arxiv.org/abs/2609.07211v1
• Date: Mon, 07 Sep 2026 08:29:59 GMT
• Title: Phase-and-First-Arrival VLM Feedback for Sparse-Reward Reinforcement Learning in Surgical Manipulation
• Abstract summary: In sparse-reward reinforcement learning, terminal rewards collapse such attempts to the same outcome.<n>We introduce phase-and-first-arrival feedback: one VLM query per recorded episode identifies the furthest visually verified task phase.<n>Across five simulated tasks, our method reaches 75.2% mean success, compared with 52.1% for a reward based on Contrastive Language-Image Pre-training.
• Score: 18.513281440175113
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse outcome feedback limits what robots can learn from unsuccessful attempts at complex manipulation. Failed multi-stage surgical attempts can contain grasps, lifts, or transfers worth reusing. In sparse-reward reinforcement learning, terminal rewards collapse such attempts to the same outcome, while scalar vision-language model (VLM) ratings reveal neither what progress merits credit nor when it occurred. We introduce phase-and-first-arrival feedback: one VLM query per recorded episode identifies the furthest visually verified task phase and when that phase is first reached, allowing the learner to reuse partial behavior and localize credit. We instantiate it in SurgPhaseBench, a phase-structured suite spanning rigid and deformable tasks, and evaluate it in simulation and hardware. Across five simulated tasks, our method reaches 75.2% mean success, compared with 52.1% for a reward based on Contrastive Language-Image Pre-training (CLIP) using the same visual input; the advantage persists when only the feedback representation changes. On hardware, the same record supports autonomous block picking and slip recovery. Together, these results show that trajectory-level visual supervision can preserve partial progress while providing the temporal credit needed for sparse-reward control.

Related papers

• Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation [51.43645327777772]
  We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations.<n>It penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress.<n>Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance.
  arXiv  Detail & Related papers  (2026-08-19T10:44:20Z)
• DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation [67.08835970838996]
  Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning.<n>Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels.<n>We introduce DenseReward, a dense robotic reward model that addresses both challenges.
  arXiv  Detail & Related papers  (2026-07-14T17:59:29Z)
• Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation [13.13689771374717]
  We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time.<n>The policy is trained using camera poses anchored to actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act.
  arXiv  Detail & Related papers  (2026-06-22T17:19:57Z)
• Retrospective Progress-Aware Self-Refinement for LLM Agent Training [55.2682415036601]
  RePro is a framework that trains agents to self-generate progress signals via a forward-then-reflect rollout paradigm.<n>RePro enhances the Qwen family's performance, with up to $12%$ absolute success rate gains.
  arXiv  Detail & Related papers  (2026-06-12T09:38:47Z)
• HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
  HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
  arXiv  Detail & Related papers  (2026-06-10T01:35:34Z)
• Phase-Conditioned Imitation Learning with Autonomous Failure Recovery for Robust Deformable Object Manipulation [5.524438437366922]
  We present a phase-conditioned, force-aware framework for deformable object manipulation.<n>A FiLM-conditioned ACT encoder modulates feature extraction based on the current task phase.<n>A multi-modal phase predictor fusing visual, force, and pose feedback estimates the phase in real time, detecting contact failures that are invisible to vision alone.
  arXiv  Detail & Related papers  (2026-05-28T05:59:53Z)
• See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation [59.07792608884117]
  We introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework.<n>SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure.<n>SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA.
  arXiv  Detail & Related papers  (2026-03-10T07:22:51Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning [64.08293076551601]
  We propose a novel method of using a learned measure for identifying positive pairs. Our Retrieval-Based Reconstruction measure measures the similarity between two sequences. We show that the REBAR error is a predictor of mutual class membership.
  arXiv  Detail & Related papers  (2023-11-01T13:44:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.