Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation

• URL: http://arxiv.org/abs/2606.24633v1
• Date: Tue, 23 Jun 2026 14:27:10 GMT
• Title: Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation
• Abstract summary: We propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations.<n>ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning.<n> Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines.
• Score: 38.88183703128359
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human demonstrations for robot imitation learning often contain mistakes and corrective behaviors, such as imprecise grasps, object misalignment, unstable contact, and repeated attempts. While these segments are commonly treated as noisy or suboptimal data, they provide valuable evidence about when execution deviates from a desirable path and how task feasibility can be restored. However, existing reward and value models often rely on monotonic progress assumptions, which capture coarse task advancement but may overlook local execution errors and corrective behaviors in imperfect demonstrations. In this work, we propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations by leveraging retry events as sparse supervision. ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning induced by sparsely annotated retry keypoints. The learned value model is then used to reweight demonstration chunks for downstream behavior cloning, reducing the influence of harmful execution errors while preserving useful corrective behaviors. Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines and improves imitation learning from imperfect demonstrations.

Related papers

• RoboDrop: Curating VLA Post-Training Data via Local Gradient Compatibility [17.469738521824905]
  We introduce textscRoboDrop, a data-curation framework that audits supervision using local gradient compatibility measured along the training trajectory.<n>We evaluate RoboDrop on controlled observation--action corruptions, naturally suboptimal demonstrations in simulation, and real-robot datasets containing non-expert collection errors.
  arXiv  Detail & Related papers  (2026-09-09T10:55:10Z)
• FLARE: A Failure-Aware Framework for Autonomous Correction and Recovery in Visual-Language Robotic Manipulation [43.89036551517797]
  Vision-Language-Action Models (VLAs) have demonstrated significant promise in generalizing to complex, long-horizon robotic manipulation tasks.<n>Their performance remains brittle, as they are typically trained on trajectory-monotonic, failure-free demonstrations.<n>We propose FLARE, a novel framework that endows VLAs with robust error recovery capabilities through a Retry" and Reset" paradigm.
  arXiv  Detail & Related papers  (2026-08-27T05:58:06Z)
• Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation [51.43645327777772]
  We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations.<n>It penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress.<n>Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance.
  arXiv  Detail & Related papers  (2026-08-19T10:44:20Z)
• DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation [67.08835970838996]
  Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning.<n>Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels.<n>We introduce DenseReward, a dense robotic reward model that addresses both challenges.
  arXiv  Detail & Related papers  (2026-07-14T17:59:29Z)
• World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays [20.499876853160206]
  World Action Models (WAMs) can generate future visual observations.<n>We build on this generative capability to propose Recurrent Generative Replay (REGEN)<n>REGEN synthesizes pseudo-replay trajectories conditioned only on prior task instructions and current-task observations.
  arXiv  Detail & Related papers  (2026-06-25T17:59:56Z)
• Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR [23.784808278172164]
  We study the impact of systematic verification errors onReinforcement Learning with Verifiable Rewards (RLVR)<n>Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes.
  arXiv  Detail & Related papers  (2026-04-06T15:02:52Z)
• Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis [21.197844940385725]
  Current failure-learning paradigms rely on either costly and unsafe real-world data collection or simulator-based perturbations.<n>We introduce Dream2Fix, a framework that synthesizes photorealistic, counterfactual failure rollouts directly from successful real-world demonstrations.<n>By perturbing actions within a generative world model, Dream2Fix creates paired failure-language data without relying on simulators.
  arXiv  Detail & Related papers  (2026-03-13T19:02:58Z)
• Attribution-Guided Model Rectification of Unreliable Neural Network Behaviors [60.06461883533697]
  We leverage rank-one model editing to establish an attribution-guided model rectification framework.<n>We first distinguish our rectification setting from existing model editing, yielding a formulation that corrects unreliable behavior.<n>To target the primary source of misbehavior, we introduce an attribution-guided layer localization method.
  arXiv  Detail & Related papers  (2026-03-08T01:06:07Z)
• Reshaping Action Error Distributions for Reliable Vision-Language-Action Models [69.38615670891038]
  In robotic manipulation, vision-language-action (VLA) models have emerged as a promising paradigm for learning generalizable and scalable robot policies.<n>We focus on continuous-action VLA models and move beyond conventional MSE-based regression by reshaping action error distributions during training.<n>We evaluate our approaches across standard, few-shot, and noisy settings on multiple representative VLA architectures.
  arXiv  Detail & Related papers  (2026-02-04T05:37:09Z)
• Learning from Demonstrations via Capability-Aware Goal Sampling [12.442790487354742]
  Cago is a learning-from-demonstrations method that mitigates the brittle dependence on expert trajectories for direct imitation.<n>We show that Cago significantly improves sample efficiency and final performance across a range of sparse-reward, goal-conditioned tasks.
  arXiv  Detail & Related papers  (2026-01-13T17:03:31Z)
• CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding [60.06899554269808]
  CycleVLA is a system that equips Vision-Language-Action models (VLAs) with proactive self-correction.<n>CycleVLA achieves this by integrating a progress-aware VLA that flags critical subtask transition points.<n>Extensive experiments show that CycleVLA improves performance for both well-trained and under-trained VLAs.
  arXiv  Detail & Related papers  (2026-01-05T17:31:01Z)
• Beyond Observations: Reconstruction Error-Guided Irregularly Sampled Time Series Representation Learning [38.869433924831156]
  iTimER is a self-supervised framework for ISTS representation learning.<n>It transforms unobserved timestamps into noise-aware training targets, enabling meaningful reconstruction signals.<n>iTimER consistently outperforms state-of-the-art methods under the ISTS setting.
  arXiv  Detail & Related papers  (2025-11-10T08:53:10Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.