Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2606.02132v2
• Date: Tue, 02 Jun 2026 07:53:40 GMT
• Title: Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning
• Abstract summary: Agentic reinforcement learning can induce tool abuse, where models overuse external tools even for solvable queries by internal reasoning.<n>We propose EAPO, an Efficient Agentic Policy Optimization framework that learns selective tool use.<n>Compared with GRPO, EAPO improves average performance by 10.45%, 7.27%, and 9.69%, while reducing average tool calls by 18.33%, 18.33%, and 24.59%, respectively.
• Score: 26.34952204312613
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic reinforcement learning can induce tool abuse, where models overuse external tools even for queries solvable by internal reasoning. Existing approaches mitigate this issue with uniform tool-use penalties or hard limits, which reduce tool frequency but may also suppress useful tool-assisted exploration. We propose EAPO, an Efficient Agentic Policy Optimization framework that learns selective tool use. EAPO introduces tool-free trajectories into each rollout group, applies difficulty-aware reward shaping to penalize redundant tool calls mainly on easier queries, and uses confidence-aware token reweighting to improve policy learning. Across nine mathematical and knowledge-intensive reasoning benchmarks, EAPO consistently improves the accuracy efficiency trade-off on Qwen2.5-3B, Qwen2.5-7B, and Llama3.1-8B. Compared with GRPO, EAPO improves average performance by 10.45%, 7.27%, and 9.69%, while reducing average tool calls by 18.33%, 18.33%, and 24.59%, respectively. These results show that agents can learn when not to use tools without compromising tool-integrated reasoning.

Related papers

• Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act [38.40533345940731]
  Large language model (LLM) agents increasingly interleave natural language reasoning with external tools.<n>We study when and why RL-trained agents learn shortcut tool-selection policies.
  arXiv  Detail & Related papers  (2026-09-14T19:31:04Z)
• Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning [8.739632908951696]
  We study calculator tool calling for improving mathematical reasoning on the Countdown task.<n>We apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO.<n>Results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k.
  arXiv  Detail & Related papers  (2026-08-28T15:35:03Z)
• Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agents [54.050506620820784]
  Agentic Tool Unlearning (ATU) aims to reduce both parametric recall and tool-mediated recovery while preserving normal tool use for retained knowledge.<n>ATU achieves a better balance between target forgetting and retained utility, making unlearning more robust under tool-augmented agent deployment.
  arXiv  Detail & Related papers  (2026-08-21T18:34:48Z)
• Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning [59.74608632210439]
  We show how to inject natural tool-use behavior into a strong thinking model without sacrificing its no-tool reasoning ability.<n>Our recipe yields models that achieve state-of-the-art performance in a wide range of benchmarks among open-source models.
  arXiv  Detail & Related papers  (2026-05-07T14:23:21Z)
• Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents [54.18201810286764]
  Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering.<n>In long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance.<n>We propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior.
  arXiv  Detail & Related papers  (2026-02-02T12:52:14Z)
• AdaTIR: Adaptive Tool-Integrated Reasoning via Difficulty-Aware Policy Optimization [0.0]
  Current agents exhibit cognitive offloading, redundantly invoking external tools even for simple tasks.<n>We propose a framework that shifts the paradigm from static tool invocation to difficulty-aware reasoning internalization.<n>AdaTIR successfully internalizes reasoning, outperforming baselines by 4.8% on AIME 2024 even when tool access is strictly disabled.
  arXiv  Detail & Related papers  (2026-01-21T06:18:46Z)
• Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning [63.2198957755528]
  We propose Tool-MVR, a novel Tool-Augmented LLM that achieves comprehensive System 2 reasoning through two key innovations.<n>Specifically, we first introduce Multi-Agent Meta-Verification (MAMV), a systematic pipeline that rigorously validates APIs, queries, and reasoning trajectories.<n>Second, we propose Exploration-based Reflection Learning (EXPLORE), which enhances tool reflection capabilities by leveraging tool feedback.
  arXiv  Detail & Related papers  (2025-06-05T04:35:49Z)
• Acting Less is Reasoning More! Teaching Model to Act Efficiently [87.28134636548705]
  Tool-integrated reasoning augments large language models with the ability to invoke external tools to solve tasks.<n>Current approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use.<n>We propose a framework that encourages models to produce accurate answers with minimal tool calls.<n>Our approach reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4%, while maintaining comparable answer accuracy.
  arXiv  Detail & Related papers  (2025-04-21T05:40:05Z)
• SMART: Self-Aware Agent for Tool Overuse Mitigation [58.748554080273585]
  Current Large Language Model (LLM) agents demonstrate strong reasoning and tool use capabilities, but often lack self-awareness.<n>This imbalance leads to Tool Overuse, where models unnecessarily rely on external tools for tasks with parametric knowledge.<n>We introduce SMART (Strategic Model-Aware Reasoning with Tools), a paradigm that enhances an agent's self-awareness to optimize task handling and reduce tool overuse.
  arXiv  Detail & Related papers  (2025-02-17T04:50:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.