Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

• URL: http://arxiv.org/abs/2609.16268v1
• Date: Mon, 14 Sep 2026 19:31:04 GMT
• Title: Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
• Abstract summary: Large language model (LLM) agents increasingly interleave natural language reasoning with external tools.<n>We study when and why RL-trained agents learn shortcut tool-selection policies.
• Score: 38.40533345940731
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These tool-use policies are often optimized via reinforcement learning (RL), which can amplify spurious correlations in the training data. In this work, we study when and why RL-trained agents learn shortcut tool-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39 percent. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence, rather than dataset imbalance alone, is a key factor in shortcut learning. A swapped-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect. To mitigate these failures, we introduce a dense, decision-level reward in which an LLM judge evaluates the necessity of each tool call. This tool-necessity reward effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.

Related papers

• Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agents [54.050506620820784]
  Agentic Tool Unlearning (ATU) aims to reduce both parametric recall and tool-mediated recovery while preserving normal tool use for retained knowledge.<n>ATU achieves a better balance between target forgetting and retained utility, making unlearning more robust under tool-augmented agent deployment.
  arXiv  Detail & Related papers  (2026-08-21T18:34:48Z)
• Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains [20.11899219087138]
  We study two representative thinking with images'' agents, Thyme and DeepEyesV2, across real-world understanding, OCR, chart understanding, and mathematical reasoning.<n>We find that tool access yields little consistent aggregate improvement, does not reliably reduce generated-token cost, and leaves only a small tool-only solved set.
  arXiv  Detail & Related papers  (2026-06-01T15:04:25Z)
• AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning [66.24374176797075]
  We introduce textbfAdaReasoner, a family of multimodal models that learn tool use as a general reasoning skill rather than as tool-specific or explicitly supervised behavior.<n>AdaReasoner is enabled by (i) a scalable data curation pipeline exposing models to long-horizon, multi-step tool interactions; (ii) Tool-GRPO, a reinforcement learning algorithm that prioritizes tool selection and sequencing based on end-task success; and (iii) an adaptive learning mechanism that dynamically regulates tool usage.
  arXiv  Detail & Related papers  (2026-01-26T16:04:43Z)
• Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving [26.413753656936688]
  Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation.<n>While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial.
  arXiv  Detail & Related papers  (2025-05-12T17:23:34Z)
• Acting Less is Reasoning More! Teaching Model to Act Efficiently [87.28134636548705]
  Tool-integrated reasoning augments large language models with the ability to invoke external tools to solve tasks.<n>Current approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use.<n>We propose a framework that encourages models to produce accurate answers with minimal tool calls.<n>Our approach reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4%, while maintaining comparable answer accuracy.
  arXiv  Detail & Related papers  (2025-04-21T05:40:05Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use.<n>MeCo quantifies metacognitive scores by capturing high-level cognitive signals in the representation space.<n>MeCo is fine-tuning-free and incurs minimal cost.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• Efficient Tool Use with Chain-of-Abstraction Reasoning [63.08202389132155]
  Large language models (LLMs) need to ground their reasoning to real-world knowledge.<n>There remains challenges for fine-tuning LLM agents to invoke tools in multi-step reasoning problems.<n>We propose a new method for LLMs to better leverage tools in multi-step reasoning.
  arXiv  Detail & Related papers  (2024-01-30T21:53:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.