Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring

• URL: http://arxiv.org/abs/2605.30834v1
• Date: Fri, 29 May 2026 04:40:12 GMT
• Title: Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring
• Abstract summary: textbf-and-Seek is a framework that formulates VLA failure detection as a coarsely supervised learning problem.<n>We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies.
• Score: 35.34120853605602
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose \textbf{Hide-and-Seek}, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, $π_0$, and $π_{0.5}$.Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.

Related papers

• FailureSpot: Label-Efficient Timestamp-Level Failure Detection for Vision-Language-Action Models [12.030216781017558]
  Vision-language--action (VLA) policies can fail unpredictably during long-horizon execution.<n>We propose a data-efficient framework that first leverages unlabeled VLA action chunks to construct weak supervision signals.<n>We then use active learning to select only the most uncertain trajectories for timestamp-level annotation.
  arXiv  Detail & Related papers  (2026-09-03T00:04:06Z)
• TrustVLA: Mechanism-Guided Inference-Time Defense Against Vision-Language-Action Backdoors [54.12833308568015]
  Vision-Language-Action (VLA) models are deployed through pipelines that end users cannot audit.<n>A small visual trigger redirects a long-horizon robot policy before any failure becomes observable.<n>We study this gap through two independently proposed VLA attacks from groups with distinct injection strategies.
  arXiv  Detail & Related papers  (2026-07-14T09:43:52Z)
• Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents [58.49879338545782]
  Long-horizon tasks are common in real-world robotic deployments, yet failure detection for such tasks remains underexplored.<n>We present Foresight, a failure detection framework that monitors manipulation trajectories using latent representations from an action-conditioned world model.<n>Our results suggest that action-conditioned world-model embeddings provide a scalable representation for reliable failure monitoring in long-horizon manipulation.
  arXiv  Detail & Related papers  (2026-06-22T09:32:28Z)
• VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models [19.137291311347788]
  Vision-language--action models (VLAs) achieve state-of-the-art performance on many robotic tasks, yet they can still behave unpredictably in out-of-distribution scenarios.<n>We propose VLA-FAIL, a framework that combines two novel failure detectors with minimal overhead, without requiring failure data.
  arXiv  Detail & Related papers  (2026-06-19T12:51:21Z)
• Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models [32.87104367896446]
  We propose an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion- and flow-based VLA policies.<n>AFIL uses a pretrained VLA to generate failure rollouts online, avoiding the need for handcrafted failure-mode design.<n>It then jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common vision-language backbone.
  arXiv  Detail & Related papers  (2026-05-08T19:57:11Z)
• Failure Identification in Imitation Learning Via Statistical and Semantic Filtering [2.2515832864485024]
  Imitation learning (IL) policies in robotics deliver strong performance in controlled settings but remain brittle in real-world deployments.<n>We introduce FIDeL (Failure Identification in Demonstration Learning), a policy-independent failure detection module.<n>We also introduce BotFails, a multimodal dataset of real-world tasks for failure detection in robotics.
  arXiv  Detail & Related papers  (2026-04-15T12:27:32Z)
• Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming [64.48633529149579]
  We propose a novel framework to expose the vulnerabilities of Vision-Language-Action (VLA) models against linguistic variations.<n>Our method reduces the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents.
  arXiv  Detail & Related papers  (2026-04-07T08:43:36Z)
• TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models [59.13964209628383]
  Vision-Language--Action (VLA) policies have shown strong progress in mapping language instructions and visual observations to robotic actions, yet their reliability degrades in cluttered scenes with distractors.<n>We propose TAG (Target-Agnostic Guidance), a simple inference-time guidance mechanism that explicitly reduces distractor- and appearance-induced bias in VLA policies.<n>We evaluate TAG on standard manipulation benchmarks, including LIBERO, LIBERO-Plus, and VLABench, where it consistently improves robustness under clutter and reduces near-miss and wrong-object executions.
  arXiv  Detail & Related papers  (2026-03-25T17:56:32Z)
• From Knowing to Doing Precisely: A General Self-Correction and Termination Framework for VLA models [42.553888395227766]
  We propose a lightweight, training-free framework for vision-language-action (VLA) models.<n>VLA-SCT operates as a self-correcting control loop, combining data-driven action refinement with conditional logic for termination.<n>Our method achieves consistent improvements across all datasets in the LIBERO benchmark.
  arXiv  Detail & Related papers  (2026-02-02T08:44:40Z)
• CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding [60.06899554269808]
  CycleVLA is a system that equips Vision-Language-Action models (VLAs) with proactive self-correction.<n>CycleVLA achieves this by integrating a progress-aware VLA that flags critical subtask transition points.<n>Extensive experiments show that CycleVLA improves performance for both well-trained and under-trained VLAs.
  arXiv  Detail & Related papers  (2026-01-05T17:31:01Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection [56.66677293607114]
  We propose Code-as-Monitor (CaM) for both open-set reactive and proactive failure detection.<n>To enhance the accuracy and efficiency of monitoring, we introduce constraint elements that abstract constraint-related entities.<n>Experiments show that CaM achieves a 28.7% higher success rate and reduces execution time by 31.8% under severe disturbances.
  arXiv  Detail & Related papers  (2024-12-05T18:58:27Z)
• Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress [31.952925824381325]
  We propose a runtime monitoring framework that splits the detection of failures into two complementary categories. We use Vision Language Models (VLMs) to detect when the policy confidently and consistently takes actions that do not solve the task. By unifying temporal consistency detection and VLM runtime monitoring, Sentinel detects 18% more failures than using either of the two detectors alone.
  arXiv  Detail & Related papers  (2024-10-06T22:13:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.