Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation

• URL: http://arxiv.org/abs/2603.25001v1
• Date: Thu, 26 Mar 2026 04:02:23 GMT
• Title: Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation
• Abstract summary: Multi-agent systems (MAS) often admit multiple plausible attributions due to complex inter-agent dependencies and ambiguous execution trajectories.<n>We revisit MAS failure attribution from a multi-perspective standpoint and propose multi-perspective failure attribution.<n>We introduce MP-Bench, the first benchmark designed for multi-perspective failure attribution in MAS, along with a new evaluation protocol tailored to this paradigm.
• Score: 29.37899008277516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Failure attribution is essential for diagnosing and improving multi-agent systems (MAS), yet existing benchmarks and methods largely assume a single deterministic root cause for each failure. In practice, MAS failures often admit multiple plausible attributions due to complex inter-agent dependencies and ambiguous execution trajectories. We revisit MAS failure attribution from a multi-perspective standpoint and propose multi-perspective failure attribution, a practical paradigm that explicitly accounts for attribution ambiguity. To support this setting, we introduce MP-Bench, the first benchmark designed for multi-perspective failure attribution in MAS, along with a new evaluation protocol tailored to this paradigm. Through extensive experiments, we find that prior conclusions suggesting LLMs struggle with failure attribution are largely driven by limitations in existing benchmark designs. Our results highlight the necessity of multi-perspective benchmarks and evaluation protocols for realistic and reliable MAS debugging.

Related papers

• VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems [79.51005192758262]
  Large language model-driven multi-agent systems excel at complex tasks.<n>However, unreliable agents remain a key bottleneck to system-level reliability.<n>We propose a hypothesis verification framework for agent failure attribution.
  arXiv  Detail & Related papers  (2026-05-17T14:09:35Z)
• Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agents [23.69429098915734]
  Verifier-Guided Action Selection (VegAS) is a test-time framework designed to improve robustness of MLLM-based embodied agents.<n>We show that VeGAS consistently improves generalization, achieving up to a 36% relative performance gain over strong CoT baselines.
  arXiv  Detail & Related papers  (2026-05-12T18:08:24Z)
• Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation [49.86232017439639]
  We introduce Multi-Output Augmented Behavioral Cloning (MA-BC)<n>MA-BC partitions divergent expert data while pooling state-action pairs where no behavior conflict is observed.<n>We establish a novel lower bound for multi-objective imitation learning, demonstrating that MA-BC is minimax optimal.
  arXiv  Detail & Related papers  (2026-05-12T11:49:08Z)
• MAVEN: Multi-Agent Verification-Elaboration Network with In-Step Epistemic Auditing [18.35295672031847]
  MAVEN is a framework designed to transform LLMs into deliberate reasoners through explicit role-decoupling.<n>MAVEN consistently outperforms latent reasoning models such as GEMINI-3.1-Pro.<n>MAVEN is fully model-agnostic, serving as a strong and transferable reasoning booster.
  arXiv  Detail & Related papers  (2026-05-08T12:11:08Z)
• Multimodal Fact-Level Attribution for Verifiable Reasoning [80.60864342985748]
  Multimodal large language models (MLLMs) are increasingly used for real-world tasks involving multi-step reasoning and long-form generation.<n>Existing multimodal grounding benchmarks and evaluation methods fail to assess attribution in complex multimodal reasoning.<n>We introduce MuRGAt, a benchmark for evaluating fact-level multimodal attribution in settings that require reasoning beyond direct observation.
  arXiv  Detail & Related papers  (2026-02-12T03:10:02Z)
• MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems [59.20800753428596]
  We present MAS-ProVe, a systematic empirical study of process verification for multi-agent systems (MAS)<n>Our study spans three verification paradigms (LLM-as-a-Judge, reward models, and process reward models)<n>We find that process-level verification does not consistently improve performance and frequently exhibits high variance.
  arXiv  Detail & Related papers  (2026-02-03T03:30:36Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attribution [0.7226144684379191]
  We present ECHO, a novel algorithm that combines hierarchical context representation, objective analysis-based evaluation, and consensus voting to improve error attribution accuracy.<n> Experimental results demonstrate that ECHO outperforms existing methods across various multi-agent interaction scenarios.
  arXiv  Detail & Related papers  (2025-10-06T15:07:13Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration [63.31211701741323]
  We extend multi-agent multi-model reasoning to generation, specifically to improving faithfulness through refinement.<n>We design intrinsic evaluations for each subtask, with our findings indicating that both multi-agent (multiple instances) and multi-model (diverse LLM types) approaches benefit error detection and critiquing.<n>We consolidate these insights into a final "recipe" called Multi-Agent Multi-Model Refinement (MAMM-Refine), where multi-agent and multi-model collaboration significantly boosts performance.
  arXiv  Detail & Related papers  (2025-03-19T14:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.