What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom

• URL: http://arxiv.org/abs/2602.01334v1
• Date: Sun, 01 Feb 2026 17:00:50 GMT
• Title: What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom
• Abstract summary: We introduce MED (Measure-Explain-Diagnose) to disentangle intrinsic capability changes from tool-induced effects.<n>We find that improvements are dominated by intrinsic learning, while tool-use RL mainly reduces tool-induced harm.<n>Overall, current vision tool-use RL learns to coexist safely with tools rather than master them.
• Score: 26.940122024948597
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision tool-use reinforcement learning (RL) can equip vision-language models with visual operators such as crop-and-zoom and achieves strong performance gains, yet it remains unclear whether these gains are driven by improvements in tool use or evolving intrinsic capabilities.We introduce MED (Measure-Explain-Diagnose), a coarse-to-fine framework that disentangles intrinsic capability changes from tool-induced effects, decomposes the tool-induced performance difference into gain and harm terms, and probes the mechanisms driving their evolution. Across checkpoint-level analyses on two VLMs with different tool priors and six benchmarks, we find that improvements are dominated by intrinsic learning, while tool-use RL mainly reduces tool-induced harm (e.g., fewer call-induced errors and weaker tool schema interference) and yields limited progress in tool-based correction of intrinsic failures. Overall, current vision tool-use RL learns to coexist safely with tools rather than master them.

Related papers

• Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs [35.17881745040865]
  Tool-augmented multimodal reasoning integrates external tools into multimodal large language models.<n>We propose ReVISE, a framework that equips MLLMs with verification and dynamic error recovery for tool-augmented reasoning.
  arXiv  Detail & Related papers  (2026-08-29T17:22:08Z)
• Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agents [54.050506620820784]
  Agentic Tool Unlearning (ATU) aims to reduce both parametric recall and tool-mediated recovery while preserving normal tool use for retained knowledge.<n>ATU achieves a better balance between target forgetting and retained utility, making unlearning more robust under tool-augmented agent deployment.
  arXiv  Detail & Related papers  (2026-08-21T18:34:48Z)
• Beyond the Eye: Efficient Multimodal Reasoning via Self-Regulated Implicit Visual Tools [65.7098588070281]
  We propose Beyond the Eye (BEE), a novel implicit visual tool paradigm centered on self-regulated capability.<n>BEE incorporates visual tool invocation behaviors into the training objective and encourages the model to develop a self-regulated invocation mechanism.<n>BEE achieves state-of-the-art performance in fine-grained visual perception while remaining competitive in general reasoning tasks.
  arXiv  Detail & Related papers  (2026-07-13T05:32:00Z)
• Visual Reasoning through Tool-supervised Reinforcement Learning [18.76805336754779]
  We propose a novel Tool-supervised Reinforcement Learning (ToolsRL) framework, with direct tool supervision for more effective tool-use learning.<n>We focus on a series of simple, native, and interpretable visual tools, including zoom-in, rotate, flip, and draw point/line.<n>A reinforcement learning curriculum is developed, where the first stage is solely optimized by a set of well motivated tool-specific rewards, and the second stage is trained with the accuracy targeted rewards while allowing calling tools.
  arXiv  Detail & Related papers  (2026-04-21T19:48:19Z)
• ForgeryVCR: Visual-Centric Reasoning via Efficient Forensic Tools in MLLMs for Image Forgery Detection and Localization [62.03035862528452]
  ForgeryVCR is a framework that materializes imperceptible traces into explicit visual intermediates via Visual-Centric Reasoning.<n>ForgeryVCR achieves state-of-the-art (SOTA) performance in both detection and localization tasks.
  arXiv  Detail & Related papers  (2026-02-15T11:14:47Z)
• AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning [66.24374176797075]
  We introduce textbfAdaReasoner, a family of multimodal models that learn tool use as a general reasoning skill rather than as tool-specific or explicitly supervised behavior.<n>AdaReasoner is enabled by (i) a scalable data curation pipeline exposing models to long-horizon, multi-step tool interactions; (ii) Tool-GRPO, a reinforcement learning algorithm that prioritizes tool selection and sequencing based on end-task success; and (iii) an adaptive learning mechanism that dynamically regulates tool usage.
  arXiv  Detail & Related papers  (2026-01-26T16:04:43Z)
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments [70.42705564227548]
  We propose an automated environment construction pipeline for large language models (LLMs)<n>This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools.<n>We also introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution.
  arXiv  Detail & Related papers  (2025-08-12T09:45:19Z)
• Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning [63.2198957755528]
  We propose Tool-MVR, a novel Tool-Augmented LLM that achieves comprehensive System 2 reasoning through two key innovations.<n>Specifically, we first introduce Multi-Agent Meta-Verification (MAMV), a systematic pipeline that rigorously validates APIs, queries, and reasoning trajectories.<n>Second, we propose Exploration-based Reflection Learning (EXPLORE), which enhances tool reflection capabilities by leveraging tool feedback.
  arXiv  Detail & Related papers  (2025-06-05T04:35:49Z)
• ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolution [77.86222359025011]
  We propose ToolACE-DEV, a self-improving framework for tool learning.<n>First, we decompose the tool-learning objective into sub-tasks that enhance basic tool-making and tool-using abilities.<n>We then introduce a self-evolving paradigm that allows lightweight models to self-improve, reducing reliance on advanced LLMs.
  arXiv  Detail & Related papers  (2025-05-12T12:48:30Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use.<n>MeCo quantifies metacognitive scores by capturing high-level cognitive signals in the representation space.<n>MeCo is fine-tuning-free and incurs minimal cost.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• Reducing Tool Hallucination via Reliability Alignment [31.761771794788462]
  Large Language Models (LLMs) have expanded their capabilities beyond language generation to interact with external tools, enabling automation and real-world applications.<n>Tool hallucinations, where models either select inappropriate tools or misuse them, pose significant challenges, leading to erroneous task execution, increased computational costs, and reduced system reliability.<n>We introduce RelyToolBench, which integrates specialized test cases and novel metrics to assess hallucination-aware task success and efficiency.<n>Finally, we propose Relign, a reliability alignment framework that expands the tool-use action space to include indecisive actions, allowing LLMs to defer tool use, seek clarification, or adjust tool selection
  arXiv  Detail & Related papers  (2024-12-05T13:10:54Z)
• CITI: Enhancing Tool Utilizing Ability in Large Language Models without Sacrificing General Performance [17.723293304671877]
  We propose a Component-based Tool-utilizing ability Injection method (CITI) According to the gradient-based importance score of different components, CITI alleviates the capability conflicts caused by fine-tuning process. Experimental results demonstrate that our approach achieves outstanding performance across a range of evaluation metrics.
  arXiv  Detail & Related papers  (2024-09-20T04:06:28Z)
• What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks [33.51887014808227]
  This paper explores the impact of both internal and external factors on the performance of tool learning frameworks. We find several insightful conclusions for future work, including the observation that LLMs can benefit significantly from increased trial and exploration.
  arXiv  Detail & Related papers  (2024-07-03T11:06:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.