Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols

• URL: http://arxiv.org/abs/2512.02787v2
• Date: Wed, 03 Dec 2025 06:27:29 GMT
• Title: Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols
• Abstract summary: We release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs.<n>Based on the dataset, we establish ViFailbackBench, a benchmark 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities.<n>We build the ViFailback-8B VLM, which achieves significant overall performance improvement on ViFailback-Bench.
• Score: 20.9173397267616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project Website: https://x1nyuzhou.github.io/vifailback.github.io/

Related papers

• Sentinel-VLA: A Metacognitive VLA Model with Active Status Monitoring for Dynamic Reasoning and Error Recovery [62.75419724651416]
  We introduce textbfSentinel-VLA, a metacognitive VLA model equipped with an active sentinel'' module to monitor real-time execution status.<n>All training data is automatically generated and annotated through our designed pipeline.<n>Real-world experiments demonstrate that Sentinel-VLA boosts the task success rate by over 30% compared to the SOTA model, PI0.
  arXiv  Detail & Related papers  (2026-05-02T02:10:54Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• ViRectify: A Challenging Benchmark for Video Reasoning Correction with Multimodal Large Language Models [23.37951284612929]
  We construct a dataset of over 30K instances spanning dynamic perception, scientific reasoning, and embodied decision-making domains.<n>In ViRectify, we challenge MLLMs to perform step-wise error identification and generate rationales with key video evidence grounding.<n>In addition, we propose the trajectory evidence-driven correction framework, comprising step-wise error trajectory and reward modeling on visual evidence-grounded correction.
  arXiv  Detail & Related papers  (2025-12-01T09:05:02Z)
• Reinforced Visual Perception with Tools [66.79840157663237]
  We introduce a novel RL algorithm based on GRPO, designed to train models to reason with a suite of four visual tools.<n>We show that our method achieves state-of-the-art performance on several perception-heavy benchmarks.<n>Our ReVPT-3B and ReVPT-7B outperform the instruct models by 9.03% and 9.44% on CV-Bench.
  arXiv  Detail & Related papers  (2025-09-01T17:57:49Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models [89.44024245194315]
  We introduce a method that incorporates explicit visual chain-of-thought (CoT) reasoning into vision-language-action models (VLAs)<n>We introduce CoT-VLA, a state-of-the-art 7B VLA that can understand and generate visual and action tokens.<n>Our experimental results demonstrate that CoT-VLA achieves strong performance, outperforming the state-of-the-art VLA model by 17% in real-world manipulation tasks and 6% in simulation benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T22:23:04Z)
• MVTamperBench: Evaluating Robustness of Vision-Language Models [5.062181035021214]
  We introduce textbfMVTamperBench, a benchmark that systematically evaluates MLLM robustness against five prevalent tampering techniques.<n>MVTamperBench comprises 3.4K original videos, expanded into over 17K tampered clips covering 19 distinct video manipulation tasks.
  arXiv  Detail & Related papers  (2024-12-27T18:47:05Z)
• ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models [55.07988373824348]
  We study the visual generalization capabilities of three existing robotic foundation models.<n>Our study shows that the existing models do not exhibit robustness to visual out-of-domain scenarios.<n>We propose a gradual backbone reversal approach founded on model merging.
  arXiv  Detail & Related papers  (2024-09-23T17:47:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.