Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents

• URL: http://arxiv.org/abs/2604.25161v1
• Date: Tue, 28 Apr 2026 03:08:48 GMT
• Title: Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents
• Abstract summary: Embodied agents in safety-critical applications rely on multiple interdependent capabilities.<n>We propose a capability-oriented testing approach that enables failure detection and attribution.
• Score: 24.279895675425077
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied agents in safety-critical applications such as Vision-Language Navigation (VLN) rely on multiple interdependent capabilities (e.g., perception, memory, planning, decision), making failures difficult to localize and attribute. Existing testing methods are largely system-level and provide limited insight into which capability deficiencies cause task failures. We propose a capability-oriented testing approach that enables failure detection and attribution by combining (1) adaptive test case generation via seed selection and mutation, (2) capability oracles for identifying capability-specific errors, and (3) a feedback mechanism that attributes failures to capabilities and guides further test generation. Experiments show that our method discovers more failure cases and more accurately pinpoints capability-level deficiencies than state-of-the-art baselines, providing more interpretable and actionable guidance for improving embodied agents.

Related papers

• Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning [109.33717747526583]
  Large language models generate computationally expensive yet semantically void reasoning on beyond-capability tasks.<n>We introduce textbfCaRL (textbfCapability-textbfaligned textbfReinforcement textbfL) to align behavior with capability boundaries.<n>Experiments demonstrate a substantial reduction in futile reasoning while preserving performance across task difficulties.
  arXiv  Detail & Related papers  (2026-07-31T09:30:33Z)
• Hierarchical Fault Localization for Autonomous Driving Systems with Hypothesis Validation and Intent Analysis [14.387618464803774]
  We present HINT, a two-phase framework for hierarchical ADS fault localization based on hypothesis validation and intent analysis.<n>In Phase I, HINT transforms failure-triggering execution recordings into multi-modal abstractions.<n>In Phase II, it reconstructs design-side intent and implementation-side behavior, then localizes suspicious code through reliability-aware consistency checking.
  arXiv  Detail & Related papers  (2026-07-14T10:17:01Z)
• Can Large Language Models Generate Observability-Aware Code? [51.30941152293301]
  We present a systematic study of observability in agent-generated systems.<n>Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals at runtime.<n>We introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure.
  arXiv  Detail & Related papers  (2026-07-07T03:20:52Z)
• Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing [12.09179106162719]
  We propose Skill-RAG, a failure-aware RAG framework that couples a hidden-state prober with a prompt-based skill router.<n> Experiments show that Skill-RAG substantially improves accuracy on hard cases persisting after multi-turn retrieval.
  arXiv  Detail & Related papers  (2026-04-17T07:25:43Z)
• Agentic Uncertainty Quantification [76.94013626702183]
  We propose a unified Dual-Process Agentic UQ (AUQ) framework that transforms verbalized uncertainty into active, bi-directional control signals.<n>Our architecture comprises two complementary mechanisms: System 1 (Uncertainty-Aware Memory, UAM), which implicitly propagates verbalized confidence and semantic explanations to prevent blind decision-making; and System 2 (Uncertainty-Aware Reflection, UAR), which utilizes these explanations as rational cues to trigger targeted inference-time resolution only when necessary.
  arXiv  Detail & Related papers  (2026-01-22T07:16:26Z)
• From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models [77.04403907729738]
  This survey charts the evolution of uncertainty from a passive diagnostic metric to an active control signal guiding real-time model behavior.<n>We demonstrate how uncertainty is leveraged as an active control signal across three frontiers.<n>This survey argues that mastering the new trend of uncertainty is essential for building the next generation of scalable, reliable, and trustworthy AI.
  arXiv  Detail & Related papers  (2026-01-22T06:21:31Z)
• Feature-Aware Test Generation for Deep Learning Models [0.5368630420272898]
  We introduce Detect, a feature-aware test generation framework for vision-based deep learning (DL) models.<n>It generates inputs by perturbing disentangled semantic attributes within the latent space.<n>It identifies which features lead to behavior shifts and uses a vision-language model for semantic attribution.
  arXiv  Detail & Related papers  (2026-01-20T15:41:06Z)
• Hierarchical Vision Language Action Model Using Success and Failure Demonstrations [60.82332413442677]
  We introduce VINE, a hierarchical vision-language-action model that separates high-level reasoning from low-level control.<n>System 2 performs feasibility-guided tree search over a 2D scene-graph abstraction.<n>System 1 executes low-level actions without modifying the agent's core skills.
  arXiv  Detail & Related papers  (2025-12-03T15:58:38Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• TrustLoRA: Low-Rank Adaptation for Failure Detection under Out-of-distribution Data [62.22804234013273]
  We propose a simple failure detection framework to unify and facilitate classification with rejection under both covariate and semantic shifts.<n>Our key insight is that by separating and consolidating failure-specific reliability knowledge with low-rank adapters, we can enhance the failure detection ability effectively and flexibly.
  arXiv  Detail & Related papers  (2025-04-20T09:20:55Z)
• FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research [0.6286531904189063]
  Approaches to scaling AI supervision include debate, critique, and prover-verifier games.<n>We present FindTheFlaws, a group of five diverse datasets spanning medicine, mathematics, science, coding, and the Lojban language.<n>We evaluate frontier models' critiquing capabilities and observe a range of performance that can be leveraged for scalable oversight experiments.
  arXiv  Detail & Related papers  (2025-03-29T06:38:30Z)
• How Execution Features Relate to Failures: An Empirical Study and Diagnosis Approach [11.857060911501016]
  Fault localization aims to identify code regions likely responsible for failures.<n>Traditional techniques primarily correlate statement execution with failures.<n>We analyzed 17 execution features and assessed their correlation with failure outcomes.
  arXiv  Detail & Related papers  (2025-02-25T22:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.