Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?

• URL: http://arxiv.org/abs/2607.09996v1
• Date: Fri, 10 Jul 2026 21:45:16 GMT
• Title: Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
• Abstract summary: We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems.<n>We construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios.
• Score: 38.40404144021991
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.

Related papers

• Adaptive Influence Graphs for Failure Attribution in Multi-Agent Systems [16.78530745432555]
  We introduce Adaptive Influence Graphs (AIGs), a two-stage agentic framework that transforms a failed trace into a structured graph and then navigates it to identify the critical error.<n>Across multiple models, we show that richer trace representations consistently improve failure attribution.<n>This affirms our hypothesis that attribution depends not only on the diagnosing model, but also on how the trace is represented and explored.
  arXiv  Detail & Related papers  (2026-08-25T10:18:19Z)
• Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents [80.03824805228719]
  A major challenge in developing computer-use agents is collecting large-scale, high-quality trajectories.<n>We propose a data-centric paradigm that turns failed trajectories into agent improvements.<n>We validate this approach with the state-of-the-art OpenCUA-72B model on the OSWorld benchmark.
  arXiv  Detail & Related papers  (2026-06-30T07:44:37Z)
• Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability [8.036549927091286]
  This paper introduces a failure-aware observability framework for diagnosing wasted computation in multi-agent LLM traces.<n>We instantiate the framework in a three- agent question-answering system and evaluate it on 165 GAIA validation traces under identical execution caps.
  arXiv  Detail & Related papers  (2026-05-31T17:50:11Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? [71.21547572568655]
  AgenTracer-8B is a lightweight failure tracer trained with multi-granular reinforcement learning.<n>On the Who&When benchmark, AgenTracer-8B outperforms giant proprietary LLMs like Gemini-2.5-Pro and Claude-4-Sonnet by up to 18.18%.<n>AgenTracer-8B delivers actionable feedback to off-the-shelf multi-agent systems like MetaGPT and MaAS with 4.8-14.2% performance gains.
  arXiv  Detail & Related papers  (2025-09-03T13:42:14Z)
• Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks [8.218266805768687]
  We present a benchmark of 34 representative programmable tasks designed to rigorously assess autonomous agents.<n>We evaluate three popular open-source agent frameworks combined with two LLM backbones, observing a task completion rate of approximately 50%.<n>We develop a three-tier taxonomy of failure causes aligned with task phases, highlighting planning errors, task execution issues, and incorrect response generation.
  arXiv  Detail & Related papers  (2025-08-18T17:55:22Z)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
  Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
  arXiv  Detail & Related papers  (2025-04-30T23:09:44Z)
• AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models [95.09157454599605]
  Large Language Models (LLMs) are becoming increasingly powerful, but they still exhibit significant but subtle weaknesses.<n>Traditional benchmarking approaches cannot thoroughly pinpoint specific model deficiencies.<n>We introduce a unified framework, AutoDetect, to automatically expose weaknesses in LLMs across various tasks.
  arXiv  Detail & Related papers  (2024-06-24T15:16:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.