Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents

• URL: http://arxiv.org/abs/2606.00096v2
• Date: Tue, 02 Jun 2026 05:03:53 GMT
• Title: Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents
• Abstract summary: Visual agents employ external visual tools within visual chains of thought to incorporate fine-grained evidence.<n>In this paper, we move beyond simple visual search tasks to investigate more challenging tasks, including 3D spatial reasoning and medical visual question answering.<n>We identify a tool-use collapse phenomenon: models progressively stop using tools while still achieving higher task accuracy.<n>We find that vanilla training and tool-use encouragement both reduce rollout diversity, explaining why higher tool use does not yield stronger reasoning performance.
• Score: 8.575257729172202
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Visual agents employ external visual tools within visual chains of thought to incorporate fine-grained evidence. While prior work has mainly studied these tools in visual search tasks, their role in more complex visual reasoning remains underexplored. In this paper, we move beyond simple visual search tasks to investigate more challenging tasks, including 3D spatial reasoning and medical visual question answering, where agents must integrate tool-acquired local evidence with the global context. We identify a {tool-use collapse phenomenon: models progressively stop using tools while still achieving higher task accuracy. Moreover, we observe a clear asymmetry: (i) completely eliminating tool use degrades performance, whereas (ii) incentivizing tool use yields only marginal gains despite substantially increasing usage. We find that vanilla training and tool-use encouragement both reduce rollout diversity, explaining why higher tool use does not yield stronger reasoning performance. Motivated by these findings, we add an entropy regularization term to encourage diverse rollout exploration, achieving the best performance despite gradually declining tool usage. Overall, our findings suggest a training-time view of tools as scaffolding, where broader exploration over language generation and visual tool invocation improves reasoning despite tool-use collapse. Project page: https://scaffolded-exploration.github.io

Related papers

• ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capability [16.62013823879194]
  Tool-augmented large language model agents excel at long-horizon tasks, yet they are typically post-trained on fixed toolsets.<n>We identify the core obstacle in such toolset expansion problem as behavioral inertia.<n>We demonstrate that injecting counterfactual anchor contexts at critical decision points can break this inertia.
  arXiv  Detail & Related papers  (2026-07-14T06:03:39Z)
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• Visual Reasoning through Tool-supervised Reinforcement Learning [18.76805336754779]
  We propose a novel Tool-supervised Reinforcement Learning (ToolsRL) framework, with direct tool supervision for more effective tool-use learning.<n>We focus on a series of simple, native, and interpretable visual tools, including zoom-in, rotate, flip, and draw point/line.<n>A reinforcement learning curriculum is developed, where the first stage is solely optimized by a set of well motivated tool-specific rewards, and the second stage is trained with the accuracy targeted rewards while allowing calling tools.
  arXiv  Detail & Related papers  (2026-04-21T19:48:19Z)
• The Tool Illusion: Rethinking Tool Use in Web Agents [60.49303190288166]
  We revisit tool use in web agents through an extensive and carefully controlled study.<n>Our findings both revise some prior conclusions and complement others with broader evidence.
  arXiv  Detail & Related papers  (2026-04-03T21:18:26Z)
• AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning [66.24374176797075]
  We introduce textbfAdaReasoner, a family of multimodal models that learn tool use as a general reasoning skill rather than as tool-specific or explicitly supervised behavior.<n>AdaReasoner is enabled by (i) a scalable data curation pipeline exposing models to long-horizon, multi-step tool interactions; (ii) Tool-GRPO, a reinforcement learning algorithm that prioritizes tool selection and sequencing based on end-task success; and (iii) an adaptive learning mechanism that dynamically regulates tool usage.
  arXiv  Detail & Related papers  (2026-01-26T16:04:43Z)
• Transductive Visual Programming: Evolving Tool Libraries from Experience for Spatial Reasoning [63.071280297939005]
  We present Transductive Visual Programming (TVP), a novel framework that builds new tools from its own experience rather than speculation.<n>TVP achieves state-of-the-art performance, outperforming GPT-4o by 22% and the previous best visual programming system by 11%.<n>Our work establishes experience-driven transductive tool creation as a powerful paradigm for building self-evolving visual programming agents.
  arXiv  Detail & Related papers  (2025-12-24T04:30:21Z)
• ToolLibGen: Scalable Automatic Tool Creation and Aggregation for LLM Reasoning [80.10274552177096]
  Large Language Models (LLMs) equipped with external tools have demonstrated enhanced performance on complex reasoning tasks.<n>The widespread adoption of this tool-augmented reasoning is hindered by the scarcity of domain-specific tools.<n>We propose a systematic approach to automatically an unstructured collection of tools into a structured tool library.
  arXiv  Detail & Related papers  (2025-10-09T04:11:16Z)
• Tool Graph Retriever: Exploring Dependency Graph-based Tool Retrieval for Large Language Models [43.50789219459378]
  We propose Tool Graph Retriever (TGR), which exploits the dependencies among tools to learn better tool representations for retrieval.<n>First, we construct a dataset termed TDI300K to train a discriminator for identifying tool dependencies.<n>Then, we represent all candidate tools as a tool dependency graph and use graph convolution to integrate the dependencies into their representations.
  arXiv  Detail & Related papers  (2025-08-07T08:36:26Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use.<n>MeCo quantifies metacognitive scores by capturing high-level cognitive signals in the representation space.<n>MeCo is fine-tuning-free and incurs minimal cost.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• Enhancing Tool Retrieval with Iterative Feedback from Large Language Models [9.588592185027455]
  Large language models (LLMs) can effectively handle a certain amount of tools through in-context learning or fine-tuning. In real-world scenarios, the number of tools is typically extensive and irregularly updated, emphasizing the necessity for a dedicated tool retrieval component. We propose to enhance tool retrieval with iterative feedback from the large language model.
  arXiv  Detail & Related papers  (2024-06-25T11:12:01Z)
• What Are Tools Anyway? A Survey from the Language Model Perspective [67.18843218893416]
  Language models (LMs) are powerful yet mostly for text generation tasks. We provide a unified definition of tools as external programs used by LMs. We empirically study the efficiency of various tooling methods.
  arXiv  Detail & Related papers  (2024-03-18T17:20:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.