Agents Trust Tools Too Much: Measuring Reliance on Unreliable Tools

• URL: http://arxiv.org/abs/2609.05587v1
• Date: Fri, 04 Sep 2026 17:02:32 GMT
• Title: Agents Trust Tools Too Much: Measuring Reliance on Unreliable Tools
• Abstract summary: We evaluate how agents respond to unreliable tool returns using three tools-web search, LLM sub-agent delegation, and code execution.<n>For each tool, we corrupt its returns and measure whether agents adopt the corrupted content in their final answers.<n>Agents exhibit high levels of overtrust across all three settings.
• Score: 16.535602696436424
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing evaluations of tool-using agents primarily measure whether an agent can successfully complete diverse tasks with tools. These evaluations generally assume that tools return reliable information. However, tool returns in real-world systems can be plausible yet incorrect. We investigate how agents respond to unreliable tool returns by evaluating fourteen LLMs using three tools-web search, LLM sub-agent delegation, and code execution. For each tool, we corrupt its returns and measure whether agents adopt the corrupted content in their final answers. Agents exhibit high levels of overtrust across all three settings: the mean adoption rate exceeds one third for every tool and reaches 68.0% for web search. Analysis of reasoning traces reveals a particularly concerning failure mode: agents often recognize conflicts and even recover the correct answer internally, yet present only the corrupted answer without warning the user. To mitigate agents' overtrust in tool returns, we intervene at three levels: prompting by the user, metadata from the tool provider, and post-training by the agent builder. Although some interventions help for particular models or tools, none consistently mitigates overtrust across tools. These findings identify overtrust in unreliable tools as a serious and persistent failure mode, motivating evaluations and interventions that enable agents to validate tool outputs and transparently communicate unresolved conflicts.

Related papers

• ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection [61.4148196085256]
  We present ROPE (Routed Origin Policy Enforcement), which is anchored in a structural notion of trust.<n>Our approach admits two provable guarantees: 1) at every step of a trajectory, no value whose only origin is attacker-writable content reaches an origin-guarded parameter, and 2) no rewording of an injection changes an admission decision.
  arXiv  Detail & Related papers  (2026-08-27T01:22:14Z)
• Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses [8.002494437891077]
  We analyze tool failures at the moment a return enters the agent context.<n>We show that likelihood-based signals clearly capture incomplete returns and some direct inconsistencies.<n>Some failures that are weak under likelihood signals can still redirect the agent toward state-changing actions.
  arXiv  Detail & Related papers  (2026-08-24T00:26:57Z)
• Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agents [54.050506620820784]
  Agentic Tool Unlearning (ATU) aims to reduce both parametric recall and tool-mediated recovery while preserving normal tool use for retained knowledge.<n>ATU achieves a better balance between target forgetting and retained utility, making unlearning more robust under tool-augmented agent deployment.
  arXiv  Detail & Related papers  (2026-08-21T18:34:48Z)
• AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP [0.0]
  AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector.<n> Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges.<n>This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away.
  arXiv  Detail & Related papers  (2026-07-13T05:14:12Z)
• Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains [20.11899219087138]
  We study two representative thinking with images'' agents, Thyme and DeepEyesV2, across real-world understanding, OCR, chart understanding, and mathematical reasoning.<n>We find that tool access yields little consistent aggregate improvement, does not reliably reduce generated-token cost, and leaves only a small tool-only solved set.
  arXiv  Detail & Related papers  (2026-06-01T15:04:25Z)
• MalTool: Malicious Tool Attacks on LLM Agents [52.01975462609959]
  MalTool is a coding-LLM-based framework that synthesizes tools exhibiting specified malicious behaviors.<n>We show that MalTool is highly effective even when coding LLMs are safety-aligned.
  arXiv  Detail & Related papers  (2026-02-12T17:27:43Z)
• DeepAgent: A General Reasoning Agent with Scalable Toolsets [111.6384541877723]
  DeepAgent is an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution.<n>To address the challenges of long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories.<n>We develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens.
  arXiv  Detail & Related papers  (2025-10-24T16:24:01Z)
• InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents [60.89180545430896]
  InfoMosaic-Bench is the first benchmark dedicated to multi-source information seeking in tool-augmented agents.<n>It requires agents to combine general-purpose search with domain-specific tools.<n>This design guarantees both reliability and non-triviality.
  arXiv  Detail & Related papers  (2025-10-02T17:48:03Z)
• More Vulnerable than You Think: On the Stability of Tool-Integrated LLM Agents [24.84276066855418]
  This study investigates whether agents are vulnerable to errors throughout the entire tool invocation process.<n>We observe that agents are highly susceptible to errors at each stage and agents based on open-source models are more vulnerable than those based on proprietary models.
  arXiv  Detail & Related papers  (2025-06-27T07:13:29Z)
• ToolFuzz -- Automated Agent Tool Testing [5.174808367448261]
  ToolFuzz is designed to discover two types of errors: (1) user queries leading to tool runtime errors and (2) user queries that lead to incorrect agent responses.<n>We show that ToolFuzz identifies 20x more erroneous inputs compared to the prompt-engineering approaches.
  arXiv  Detail & Related papers  (2025-03-06T14:29:52Z)
• SMART: Self-Aware Agent for Tool Overuse Mitigation [58.748554080273585]
  Current Large Language Model (LLM) agents demonstrate strong reasoning and tool use capabilities, but often lack self-awareness.<n>This imbalance leads to Tool Overuse, where models unnecessarily rely on external tools for tasks with parametric knowledge.<n>We introduce SMART (Strategic Model-Aware Reasoning with Tools), a paradigm that enhances an agent's self-awareness to optimize task handling and reduce tool overuse.
  arXiv  Detail & Related papers  (2025-02-17T04:50:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.