When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents

• URL: http://arxiv.org/abs/2606.05806v1
• Date: Thu, 04 Jun 2026 07:38:46 GMT
• Title: When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents
• Abstract summary: Existing benchmarks evaluate Tool-Integrated Reasoning in LLMs on idealized ''happy paths''<n>We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents.
• Score: 48.32450507410869
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a $2 \times 2$ taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37\% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale $3.66\times$ slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.

Related papers

• What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity [0.9666308705700454]
  Pruning can reduce the deployment cost of large language models, but its impact on context-grounded tool calling remains poorly understood.<n>We systematically study pruning-induced degradation in smart-home tool calling across four large language models (LLMs)<n>Our results show that dense models have narrow safe pruning regions followed by sharp degradation, while MoE models tolerate substantially more pruning.
  arXiv  Detail & Related papers  (2026-09-15T17:51:48Z)
• ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents [9.426759720914623]
  ParaRecover is a process-level benchmark for evaluating error localization and recovery in multi-turn parallel tool-use agents.<n>Built upon a fine-grained taxonomy of 14 error types covering planning dependencies, tool selection, and argument matching, the benchmark comprises 10,626 instances spanning two difficulty levels.<n>Experiments reveal that even state-of-the-art models still struggle with multi-turn error propagation,implicit tool-use failures, and precise replanning.
  arXiv  Detail & Related papers  (2026-09-11T02:09:33Z)
• ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents [52.531681772560724]
  Supervised fine-tuning (SFT) is built mostly on successful trajectories and offers little signal for recovery after tool failures.<n>We introduce ReGRPO, a framework that learns reflection-guided correction in tool-using agents.<n> Experiments on GTA and GAIA show that, under the same backbone and tool suite, ReGRPO consistently outperforms strong open-source baselines.
  arXiv  Detail & Related papers  (2026-06-30T09:19:38Z)
• A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks [25.713629634281077]
  We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage.<n> TASTE selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through difficulty evolution.<n>Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability.
  arXiv  Detail & Related papers  (2026-05-27T14:45:59Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Graph-Based Self-Healing Tool Routing for Cost-Efficient LLM Agents [0.0]
  Self-Healing Router is a fault-tolerant orchestration architecture.<n>It treats most agent control-flow decisions as routing rather than reasoning.<n>Every failure is either a logged reroute or an explicit escalation, never a silent skip.
  arXiv  Detail & Related papers  (2026-03-02T07:21:15Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.