ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents

• URL: http://arxiv.org/abs/2609.12345v1
• Date: Fri, 11 Sep 2026 02:09:33 GMT
• Title: ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents
• Abstract summary: ParaRecover is a process-level benchmark for evaluating error localization and recovery in multi-turn parallel tool-use agents.<n>Built upon a fine-grained taxonomy of 14 error types covering planning dependencies, tool selection, and argument matching, the benchmark comprises 10,626 instances spanning two difficulty levels.<n>Experiments reveal that even state-of-the-art models still struggle with multi-turn error propagation,implicit tool-use failures, and precise replanning.
• Score: 9.426759720914623
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing agent benchmarks mainly evaluate final task success or tool-call correctness, providing limited insight into whether agents can reliably diagnose and recover from intermediate execution failures. This limitation becomes particularly critical in multi-turn parallel tool-use scenarios, where errors may propagate across dependent branches and trigger cascading failures. We introduce ParaRecover, a process-level benchmark for evaluating error localization and recovery in multi-turn parallel tool-use agents. Built upon a fine-grained taxonomy of 14 error types covering planning dependencies, tool selection, and argument matching, the benchmark comprises 10,626 instances spanning two difficulty levels. To enable finegrained, process-oriented evaluation, we further propose the SDE rubric, which measures structural integrity, diagnostic reasoning, and evolutionary strategy during agent execution.Experiments across more than ten mainstream LLMs reveal that even state-of-the-art models still struggle with multi-turn error propagation,implicit tool-use failures, and precise replanning. Moreover, we demonstrate that the SDE rubric provides effective supervision signals for improving agents' reflective recovery capabilities. Our data and code are available at https://github.com/gbw206/ParaRecover.

Related papers

• From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
  We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
  arXiv  Detail & Related papers  (2026-08-17T01:40:14Z)
• Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction [22.654396704495934]
  Development-set failures can recover part of the missing diagnostic substrate by deciding which recovery interventions are admissible.<n>We propose DARC, a diagnosis-guided recovery harness that profiles task-family failure modes, prunes mismatched interventions from a shared recovery library, and freezes a verifier-selected success-cost policy for deployment.
  arXiv  Detail & Related papers  (2026-08-12T08:14:45Z)
• ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents [52.531681772560724]
  Supervised fine-tuning (SFT) is built mostly on successful trajectories and offers little signal for recovery after tool failures.<n>We introduce ReGRPO, a framework that learns reflection-guided correction in tool-using agents.<n> Experiments on GTA and GAIA show that, under the same backbone and tool suite, ReGRPO consistently outperforms strong open-source baselines.
  arXiv  Detail & Related papers  (2026-06-30T09:19:38Z)
• When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents [48.32450507410869]
  Existing benchmarks evaluate Tool-Integrated Reasoning in LLMs on idealized ''happy paths''<n>We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents.
  arXiv  Detail & Related papers  (2026-06-04T07:38:46Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems [0.0]
  We present AgentTrace, a lightweight causal tracing framework for post-hoc failure diagnosis in deployed multi-agent AI systems.<n>AgentTrace reconstructs causal graphs from execution logs, traces backward from error manifestations, and ranks candidate root causes using interpretable structural and positional signals.<n>Our results suggest that causal tracing provides a practical foundation for improving the reliability and trustworthiness of agentic systems in the wild.
  arXiv  Detail & Related papers  (2026-03-16T00:46:44Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• AgentRx: Diagnosing AI Agent Failures from Execution Trajectories [9.61742219198197]
  We release a benchmark of 115 failed trajectories spanning structured API, incident management, and open-ended web/file tasks.<n>Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy.<n>We present AGENTRX, an automated domain-agnostic diagnostic framework that pinpoints the critical failure step in a failed agent trajectory.
  arXiv  Detail & Related papers  (2026-02-02T18:54:07Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.