ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents

• URL: http://arxiv.org/abs/2606.03054v1
• Date: Tue, 02 Jun 2026 02:44:27 GMT
• Title: ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents
• Abstract summary: ToolGate predicts execute/skip decisions from trajectory text and simple structural features.<n>Across two Qwen3-VL backbones, ToolGate reduces token cost to 64-69% of the unrestricted ReAct baseline.
• Score: 9.793491545168575
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tool-augmented vision-language agents can acquire external perceptual evidence through OCR, detection, segmentation, and other tools, but executing every proposed tool call is costly and sometimes unnecessary. We study the pre-call control problem: after a ReAct-style VLM agent proposes a perceptual tool call, should the call be executed, or skipped before its output enters the context? Across five benchmarks, we find that the baseline agent exhibits poor local selectivity: helpful and harmful calls occur at similar rates (11.8% vs. 9.9%), while most calls do not change the immediate forced-answer prediction. We introduce ToolGate, a lightweight external controller that predicts execute/skip decisions from trajectory text and simple structural features. Across two Qwen3-VL backbones, ToolGate reduces token cost to 64-69% of the unrestricted ReAct baseline while preserving average accuracy in cross-domain settings. With matched-domain trajectory training on Qwen3-VL-30B, it further improves average accuracy by 1.65 points. These results show that tool-augmented VLM agents benefit not only from better perceptual tools, but also from explicit control over when tool outputs are worth paying for.

Related papers

• Tunable Tool-Call Rates in LLM Agents via Representation Steering [45.332672398938136]
  We show that a tool can be controlled by a single linear direction in its residual stream, extracted without any training from the model's own tool-use preference signal.<n>We also show that the direction generalizes to unseen tools with strength comparable to each tool's own direction without favoring any specific tool choice.
  arXiv  Detail & Related papers  (2026-08-25T22:35:18Z)
• Self-Evolution for Multi-Turn Tool-Calling Agents via Divergence-Point Preference Learning [0.0]
  Multi-turn tool-using agents must coordinate long-horizon tool sequences while tracking dialogue state and policy constraints.<n>For within-benchmark self-improvement, ToolGraph combines schema-derived topology, transition weights estimated from successful rollouts, and history-aware controls for write prerequisites and repeated-search loops.<n>We then construct 161 preference pairs by locating divergence points via state-based matching and prefix-based alignment, filtered through action-correctness annotations, and train DPO under the same ToolGraph context used at inference.
  arXiv  Detail & Related papers  (2026-06-22T09:56:44Z)
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents [52.33697136670754]
  We introduce textbfHyperTool, a unified executable MCP-style tool interface that changes the model-visible unit of tool execution.<n>A model invokes HyperTool with a code block that can call existing tools through their original schemas, manipulate returned values, and pass intermediate results locally.<n>On MCP-Universe, HyperTool improves average accuracy from 15.69% to 35.29% on Qwen3-32B and from 9.93% to 33.33% on Qwen3-8B, and surpass GPT-OSS and Kimi-k2.5 on average
  arXiv  Detail & Related papers  (2026-06-11T17:56:36Z)
• Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains [20.11899219087138]
  We study two representative thinking with images'' agents, Thyme and DeepEyesV2, across real-world understanding, OCR, chart understanding, and mathematical reasoning.<n>We find that tool access yields little consistent aggregate improvement, does not reliably reduce generated-token cost, and leaves only a small tool-only solved set.
  arXiv  Detail & Related papers  (2026-06-01T15:04:25Z)
• LLM Agents Already Know When to Call Tools -- Even Without Reasoning [25.40369702634587]
  LLM agents tend to call tools indiscriminately, even when the model can answer directly.<n>No existing benchmark systematically studies when a tool call is actually needed.<n>We propose When2Tool, a benchmark of 18 environments spanning three categories of tool necessity.
  arXiv  Detail & Related papers  (2026-05-10T01:37:40Z)
• Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents [16.379755833783406]
  We show that in the presence of semantic distractors, tool-augmented reasoning does not necessarily outperform native CoT.<n>Our analysis reveals a critical tradeoff: under semantic noise, the gains from tools often fail to offset the "tool-use tax"<n>We introduce G-STEP, a lightweight inference-time gate to mitigate protocol-induced errors.
  arXiv  Detail & Related papers  (2026-04-30T18:46:01Z)
• ToolRLA: Multiplicative Reward Decomposition for Tool-Integrated Agents [1.8379860135249093]
  We present ToolRLA, a post-training pipeline for domain-specific tool agents.<n>The core contribution is a fine-grained reward function with multiplicative correctness decomposition.<n> ToolRLA achieves over three months: a 47% improvement in task completion rate.
  arXiv  Detail & Related papers  (2026-03-02T08:52:14Z)
• Budget-Aware Tool-Use Enables Effective Agent Scaling [82.6942342482552]
  Scaling test-time computation improves performance across different tasks on large language models (LLMs)<n>We study how to scale such agents effectively under explicit tool-call budgets, focusing on web search agents.<n>We introduce the Budget Tracker, a lightweight plug-in that provides the agent with continuous budget awareness.
  arXiv  Detail & Related papers  (2025-11-21T07:18:55Z)
• How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on $τ$-bench [58.114899897566964]
  In a multi-turn conversational environment, large language models (LLMs) often struggle with consistent reasoning and adherence to domain-specific policies.<n>We propose the Input-Reformulation Multi-Agent (IRMA) framework, which automatically reformulates user queries augmented with relevant domain rules.<n>IRMA significantly outperforms ReAct, Function Calling, and Self-Reflection by 16.1%, 12.7%, and 19.1%, respectively.
  arXiv  Detail & Related papers  (2025-08-28T15:57:33Z)
• Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning [63.2198957755528]
  We propose Tool-MVR, a novel Tool-Augmented LLM that achieves comprehensive System 2 reasoning through two key innovations.<n>Specifically, we first introduce Multi-Agent Meta-Verification (MAMV), a systematic pipeline that rigorously validates APIs, queries, and reasoning trajectories.<n>Second, we propose Exploration-based Reflection Learning (EXPLORE), which enhances tool reflection capabilities by leveraging tool feedback.
  arXiv  Detail & Related papers  (2025-06-05T04:35:49Z)
• Acting Less is Reasoning More! Teaching Model to Act Efficiently [87.28134636548705]
  Tool-integrated reasoning augments large language models with the ability to invoke external tools to solve tasks.<n>Current approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use.<n>We propose a framework that encourages models to produce accurate answers with minimal tool calls.<n>Our approach reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4%, while maintaining comparable answer accuracy.
  arXiv  Detail & Related papers  (2025-04-21T05:40:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.