Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models

• URL: http://arxiv.org/abs/2609.03493v1
• Date: Thu, 03 Sep 2026 07:52:56 GMT
• Title: Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models
• Abstract summary: We introduce NTEP (Necessary Tool-Evidence Path), a novel annotation scheme that explicitly specifies the essential external evidence.<n>We also propose NTEP-R (NTEP Reward), a supervision mechanism ensuring that each tool invocation strictly advances the reasoning process.<n>Our approach rewards the agent for aligning its pre-call intent with a necessary evidence-seeking goal, and for ensuring the information summarized from the post-call observation aligns with the necessary evidence.
• Score: 18.068231324319395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern vision-language models (VLMs) can directly answer many image-grounded questions, yet they often struggle with complex queries requiring fine-grained visual details or external knowledge. To acquire this missing evidence, agentic VLMs invoke tools such as image cropping, image search, and text search. However, existing training paradigms primarily evaluate tool-use based on final answer correctness, leaving evidence acquisition and utilization insufficiently supervised. This leads to two critical shortcomings: (i) models frequently issue redundant or off-target tool calls that fail to gather necessary evidence, and (ii) even when appropriate tools are called, models often fail to extract the necessary information from the resulting observations. To address these limitations, we introduce the NTEP (Necessary Tool-Evidence Path), a novel annotation scheme that explicitly specifies the essential external evidence and corresponding tool calls for each query. Building upon this, we propose NTEP-R (NTEP Reward), a supervision mechanism ensuring that each tool invocation strictly advances the reasoning process toward the final solution. Specifically, our approach rewards the agent for aligning its pre-call intent with a necessary evidence-seeking goal, and for ensuring the information summarized from the post-call observation aligns with the necessary evidence. Furthermore, we introduce a non-repeated-goal regularizer to penalize redundant calls that revisit satisfied NTEP goals. Extensive evaluations on seven image-grounded benchmarks demonstrate that our 8B-parameter instantiation, NTEP-8B, significantly improves both search-oriented accuracy and tool-use efficiency within a unified three-tool framework. These results highlight the critical value of fine-grained tool-evidence path supervision for training robust agentic VLMs.

Related papers

• Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs [35.17881745040865]
  Tool-augmented multimodal reasoning integrates external tools into multimodal large language models.<n>We propose ReVISE, a framework that equips MLLMs with verification and dynamic error recovery for tool-augmented reasoning.
  arXiv  Detail & Related papers  (2026-08-29T17:22:08Z)
• WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents [53.426679066400844]
  We introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery.<n>We develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout.<n>We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image.
  arXiv  Detail & Related papers  (2026-08-28T08:28:43Z)
• Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agents [54.050506620820784]
  Agentic Tool Unlearning (ATU) aims to reduce both parametric recall and tool-mediated recovery while preserving normal tool use for retained knowledge.<n>ATU achieves a better balance between target forgetting and retained utility, making unlearning more robust under tool-augmented agent deployment.
  arXiv  Detail & Related papers  (2026-08-21T18:34:48Z)
• FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification [41.86420087238674]
  Recent studies have revealed that agentic vision-language models often use tools unfaithfully.<n>We introduce FaithEyes, a multi-agent self-judging framework.<n>FaithEyes attains competitive or superior accuracy across visual perception and reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-07-30T13:58:08Z)
• MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts [10.608824155356377]
  We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario.<n>We observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception.<n>We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement.
  arXiv  Detail & Related papers  (2026-07-21T03:43:50Z)
• Beyond the Eye: Efficient Multimodal Reasoning via Self-Regulated Implicit Visual Tools [65.7098588070281]
  We propose Beyond the Eye (BEE), a novel implicit visual tool paradigm centered on self-regulated capability.<n>BEE incorporates visual tool invocation behaviors into the training objective and encourages the model to develop a self-regulated invocation mechanism.<n>BEE achieves state-of-the-art performance in fine-grained visual perception while remaining competitive in general reasoning tasks.
  arXiv  Detail & Related papers  (2026-07-13T05:32:00Z)
• To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling [13.42769424615184]
  We introduce a principled framework inspired by decision-making theory to evaluate web search tool-use decisions.<n>We find that models' perceived need and utility of tool calls are often misaligned with their true need and utility.<n>Our estimators enable simple controllers that can improve decision quality and lead to stronger task performance.
  arXiv  Detail & Related papers  (2026-05-01T15:38:13Z)
• RecThinker: An Agentic Framework for Tool-Augmented Reasoning in Recommendation [63.74915464611075]
  RecThinker is an agentic framework for tool-augmented reasoning in recommendation.<n>We develop a suite of specialized tools for RecThinker, enabling the model to acquire user item-side and collaborative information.
  arXiv  Detail & Related papers  (2026-03-10T16:07:17Z)
• ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning [103.7657839292775]
  ARM-Thinker is an Agentic multimodal Reward Model that autonomously invokes external tools to ground judgments in verifiable evidence.<n>We train ARM-Thinker with multi-stage reinforcement learning, jointly optimizing tool-calling decisions and judgment accuracy.<n>Our results demonstrate that agentic capabilities significantly enhance both accuracy and interpretability of reward models.
  arXiv  Detail & Related papers  (2025-12-04T18:59:52Z)
• Improving Large Language Models Function Calling and Interpretability via Guided-Structured Templates [56.73907811047611]
  Large language models (LLMs) have demonstrated strong reasoning and tool-use capabilities.<n>LLMs often fail in real-world tool-interactions due to incorrect parameterization, poor tool selection, or misinterpretation of user intent.<n>We introduce a curriculum-inspired framework that leverages structured reasoning templates to guide LLMs through more deliberate step-by-step instructions for generating function callings.
  arXiv  Detail & Related papers  (2025-09-22T17:55:14Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use.<n>MeCo quantifies metacognitive scores by capturing high-level cognitive signals in the representation space.<n>MeCo is fine-tuning-free and incurs minimal cost.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.