ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems

• URL: http://arxiv.org/abs/2604.01508v1
• Date: Thu, 02 Apr 2026 00:42:29 GMT
• Title: ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems
• Abstract summary: Tool using agents often fail for operational reasons even when language understanding is strong.<n>Common causes include invalid arguments, interface drift, weak recovery, and inefficient retry behavior.<n>We introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and recovery under explicit step, call, and retry budgets.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tool using agents often fail for operational reasons even when language understanding is strong. Common causes include invalid arguments, interface drift, weak recovery, and inefficient retry behavior. We introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and recovery under explicit step, call, and retry budgets. The benchmark covers CRUD, retrieval, file, and scheduling environments with replayable fault injection. It reports success, invalid call behavior, policy violations, recovery quality, and budgeted efficiency. We release a public dataset with 6800 tasks and a reproducible evaluation pipeline. Baseline results show fault specific recovery gains for schema aware methods, while overall success remains limited under the released authorization and hard failure settings.

Related papers

• ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents [9.426759720914623]
  ParaRecover is a process-level benchmark for evaluating error localization and recovery in multi-turn parallel tool-use agents.<n>Built upon a fine-grained taxonomy of 14 error types covering planning dependencies, tool selection, and argument matching, the benchmark comprises 10,626 instances spanning two difficulty levels.<n>Experiments reveal that even state-of-the-art models still struggle with multi-turn error propagation,implicit tool-use failures, and precise replanning.
  arXiv  Detail & Related papers  (2026-09-11T02:09:33Z)
• PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents [79.28372522980153]
  PeakBench is a benchmark of executable multi-tool dependency annotations and measured resource profiles.<n>We show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints.<n>PeakBench is a useful testbed for diagnosing resource-aware agent behavior.
  arXiv  Detail & Related papers  (2026-08-25T12:56:08Z)
• ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents [16.995082216096787]
  Large language models rely on tool calling as a fundamental agent capability.<n>Clean end-to-end (E2E) success cannot identify where a tool-use failure originates or how it propagates through a call.<n>We introduce ToolRobustBench, a stage-wise diagnostic benchmark for tool-calling agents.
  arXiv  Detail & Related papers  (2026-08-23T15:22:49Z)
• Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction [22.654396704495934]
  Development-set failures can recover part of the missing diagnostic substrate by deciding which recovery interventions are admissible.<n>We propose DARC, a diagnosis-guided recovery harness that profiles task-family failure modes, prunes mismatched interventions from a shared recovery library, and freezes a verifier-selected success-cost policy for deployment.
  arXiv  Detail & Related papers  (2026-08-12T08:14:45Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• Can LLMs Really Recover Microservice Failures? A Recovery-Aware Evaluation of Diagnosis-to-Action Reasoning [48.22526014304726]
  This paper presents R2Act, a recovery-action evaluation framework for post-diagnosis incident response.<n>R2Act defines an incident schema, quality gate, action-space representation, recovery-validity metrics, offline evaluator, and live-replay protocol.<n>We instantiate the framework as a benchmark dataset of 302 quality-audited incidents from system.
  arXiv  Detail & Related papers  (2026-07-06T03:05:17Z)
• ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents [52.531681772560724]
  Supervised fine-tuning (SFT) is built mostly on successful trajectories and offers little signal for recovery after tool failures.<n>We introduce ReGRPO, a framework that learns reflection-guided correction in tool-using agents.<n> Experiments on GTA and GAIA show that, under the same backbone and tool suite, ReGRPO consistently outperforms strong open-source baselines.
  arXiv  Detail & Related papers  (2026-06-30T09:19:38Z)
• Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability [14.862168786900382]
  We introduce ToolBench-X, a benchmark for evaluating agents under recoverable reliability hazards.<n>Starting from clean tool environments, ToolBench-X injects five structured hazard types.<n>Experiments reveal a substantial reliability gap: agents that perform well with reliable tools often fail under recoverable hazards.
  arXiv  Detail & Related papers  (2026-06-24T13:34:34Z)
• When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents [48.32450507410869]
  Existing benchmarks evaluate Tool-Integrated Reasoning in LLMs on idealized ''happy paths''<n>We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents.
  arXiv  Detail & Related papers  (2026-06-04T07:38:46Z)
• Self-Healing Agentic Orchestrators for Reliable Tool-Augmented Large Language Model Systems [0.10210859604701106]
  Self-healing agentic orchestrator treats reliability as a runtime bounded control problem.<n>Self-healing achieves 98.8% task success, compared with 94.5% for retry-only and 93.8% for full replanning.
  arXiv  Detail & Related papers  (2026-05-31T19:27:22Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• ToolMATH: A Math Tool Benchmark for Realistic Long-Horizon Multi-Tool Reasoning [11.99927786717109]
  ToolMATH turns math problems into a controlled, correctness-checkable benchmark with tool sets.<n>ToolMATH provides actionable diagnostic evidence of failure modes in tool-augmented agents.
  arXiv  Detail & Related papers  (2026-02-24T09:23:12Z)
• Robust and Efficient Tool Orchestration via Layered Execution Structures with Reflective Correction [55.13278005189741]
  We model tool orchestration as learning a layered execution structure that captures high-level tool dependencies.<n>We introduce a schema-aware reflective correction mechanism that detects and repairs errors locally.<n>This design confines errors to individual tool calls and avoids re-planning entire execution trajectories.
  arXiv  Detail & Related papers  (2026-02-21T22:20:01Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• Just-in-time Episodic Feedback Hinter: Leveraging Offline Knowledge to Improve LLM Agents Adaptation [77.90555621662345]
  We present JEF Hinter, an agentic system that distills offline traces into compact, context-aware hints.<n>A zooming mechanism highlights decisive steps in long trajectories, capturing both strategies and pitfalls.<n>Experiments on MiniWoB++, WorkArena-L1, and WebArena-Lite show that JEF Hinter consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-10-05T21:34:42Z)
• PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Cases [2.3181214107210235]
  PALADIN trains on 50,000+ recovery-annotated trajectories constructed via systematic failure injection.<n>Training uses LoRA-based fine-tuning to retain base capabilities while injecting recovery competence.<n>This approach generalizes to novel failures beyond the training distribution.
  arXiv  Detail & Related papers  (2025-09-25T10:37:30Z)
• Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions [10.598440138966028]
  Current self-reflection practices rely on prompts or one-way reasoning.<n>We propose structured reflection, which turns the path from error to repair into an explicit, controllable, and trainable action.<n>Experiments on BFCL v3 and Tool-Reflection-Bench show large gains in multi-turn tool-call success and error recovery, and a reduction of redundant calls.
  arXiv  Detail & Related papers  (2025-09-23T09:35:49Z)
• A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks [1.3398445165628463]
  This paper provides a comprehensive analysis of fault recovery performance, stability, and recovery time in a cloud-native environment. Our results indicate that Flink is the most stable and has one of the best fault recovery. K Kafka Streams shows suitable fault recovery performance and stability, but with higher event latency.
  arXiv  Detail & Related papers  (2024-04-09T10:49:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.