Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis

• URL: http://arxiv.org/abs/2603.13528v1
• Date: Fri, 13 Mar 2026 19:02:58 GMT
• Title: Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis
• Abstract summary: Current failure-learning paradigms rely on either costly and unsafe real-world data collection or simulator-based perturbations.<n>We introduce Dream2Fix, a framework that synthesizes photorealistic, counterfactual failure rollouts directly from successful real-world demonstrations.<n>By perturbing actions within a generative world model, Dream2Fix creates paired failure-language data without relying on simulators.
• Score: 21.197844940385725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While recent foundation models have significantly advanced robotic manipulation, these systems still struggle to autonomously recover from execution errors. Current failure-learning paradigms rely on either costly and unsafe real-world data collection or simulator-based perturbations, which introduce a severe sim-to-real gap. Furthermore, existing visual analyzers predominantly output coarse, binary diagnoses rather than the executable, trajectory-level corrections required for actual recovery. To bridge the gap between failure diagnosis and actionable recovery, we introduce Dream2Fix, a framework that synthesizes photorealistic, counterfactual failure rollouts directly from successful real-world demonstrations. By perturbing actions within a generative world model, Dream2Fix creates paired failure-correction data without relying on simulators. To ensure the generated data is physically viable for robot learning, we implement a structured verification mechanism that strictly filters rollouts for task validity, visual coherence, and kinematic safety. This engine produces a high-fidelity dataset of over 120k paired samples. Using this dataset, we fine-tune a vision-language model to jointly predict failure types and precise recovery trajectories, mapping visual anomalies directly to corrective actions. Extensive real-world robotic experiments show our approach achieves state-of-the-art correction accuracy, improving from 19.7% to 81.3% over prior baselines, and successfully enables zero-shot closed-loop failure recovery in physical deployments.

Related papers

• EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration [75.13534797264485]
  We show that egocentric human data capturing failure recovery processes provides a scalable alternative to robot teleoperation.<n>By efficiently arranging task-level failure configurations and recording short recovery segments, human operators can generate more than 10x as much valid recovery data per hour.
  arXiv  Detail & Related papers  (2026-07-22T04:44:26Z)
• DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation [67.08835970838996]
  Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning.<n>Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels.<n>We introduce DenseReward, a dense robotic reward model that addresses both challenges.
  arXiv  Detail & Related papers  (2026-07-14T17:59:29Z)
• SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics [49.771521708359955]
  SurgVista is a surgical world model that mitigates spatial interaction incoherence and temporal fidelity collapse.<n>It consistently outperforms state-of-the-art methods across visual quality, temporal consistency, and interaction fidelity.
  arXiv  Detail & Related papers  (2026-06-18T07:47:28Z)
• Foundational World Models Accurately Detect Bimanual Manipulator Failures [21.93685012734004]
  We train a probabilistic, history informed, world model within the compressed latent space of a pretrained vision foundation model.<n>The model outputs uncertainty estimates alongside its predictions that serve as non-conformity scores within a conformal prediction framework.<n>We show that our approach requires approximately one twentieth of the trainable parameters as the next-best learning-based approach.
  arXiv  Detail & Related papers  (2026-03-07T02:11:29Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• Continual Error Correction on Low-Resource Devices [33.29366524304944]
  We present a novel system enabling users to correct AI misclassifications through few-shot learning, requiring minimal computational resources and storage.<n>Our approach combines server-side foundation model training with on-device prototype-based classification, enabling efficient error correction through prototype updates rather than model retraining.<n>We demonstrate our system's effectiveness on both image classification and object detection tasks, achieving over 50% error correction in one-shot scenarios on Food-101 and Flowers-102 datasets.
  arXiv  Detail & Related papers  (2025-11-26T18:24:11Z)
• Action Flow Matching for Continual Robot Learning [54.10050120844738]
  Continual learning in robotics seeks systems that can constantly adapt to changing environments and tasks.<n>We introduce a generative framework leveraging flow matching for online robot dynamics model alignment.<n>We find that by transforming the actions themselves rather than exploring with a misaligned model, the robot collects informative data more efficiently.
  arXiv  Detail & Related papers  (2025-04-25T16:26:15Z)
• In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators [11.389756788049944]
  Testing autonomous robotic manipulators is challenging due to the complex software interactions between vision and control components. A crucial element of modern robotic manipulators is the deep learning based object detection model. We propose the MARTENS framework, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios.
  arXiv  Detail & Related papers  (2024-10-25T03:10:42Z)
• Learning to Recover from Plan Execution Errors during Robot Manipulation: A Neuro-symbolic Approach [7.768747914019512]
  We propose an approach (blending learning with symbolic search) for automated error discovery and recovery. We present an anytime version of our algorithm, where instead of recovering to the last correct state, we search for a sub-goal in the original plan.
  arXiv  Detail & Related papers  (2024-05-29T10:03:57Z)
• Real-to-Sim: Predicting Residual Errors of Robotic Systems with Sparse Data using a Learning-based Unscented Kalman Filter [65.93205328894608]
  We learn the residual errors between a dynamic and/or simulator model and the real robot. We show that with the learned residual errors, we can further close the reality gap between dynamic models, simulations, and actual hardware.
  arXiv  Detail & Related papers  (2022-09-07T15:15:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.