ToolFG: Towards Well-Grounded Fine-Grained Image Classification

• URL: http://arxiv.org/abs/2606.02518v1
• Date: Mon, 01 Jun 2026 17:27:48 GMT
• Title: ToolFG: Towards Well-Grounded Fine-Grained Image Classification
• Abstract summary: We propose textbfToolFG, the first tool-integrated MLLM-based framework tailored to fine-grained image classification.<n>ToolFG enables MLLMs to autonomously and flexibly use external tools during the reasoning process.
• Score: 35.58965837460018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-grained image classification (FGIC) has broad applications and has attracted significant research attention. In this paper, we explore a novel paradigm for solving FGIC by proposing \textbf{ToolFG}, the first tool-integrated MLLM-based framework tailored to FGIC. ToolFG enables MLLMs to autonomously and flexibly use external tools during the reasoning process, actively interact with images, and collect verifiable visual cues for distinguishing highly similar categories in a more \textit{reliable} and \textit{well-grounded} manner. To equip the model with such tool-use ability, we design a novel \textbf{MCTS-guided tool-use knowledge distillation mechanism}, which effectively mines tool-use- and FGIC-relevant knowledge from advanced proprietary MLLMs for model training. Furthermore, we propose a \textbf{model-tool co-evolution mechanism} that jointly refines the toolset and the model's tool-use policy, driving them toward a mutually adapted and FGIC-specialized state. Extensive experiments demonstrate the effectiveness of our framework.

Related papers

• ToolLIFT: Lifting Tool-Specific Trajectories into Function-Level Graphs for Generalizable Tool Planning [12.959820672254777]
  We propose ToolLIFT, a framework that lifts tool-specific trajectories into a function-level workflow graph (FWG) for generalizable tool planning.<n>To ensure reliable tool dataflow, we adopt source-gated and skill-specific rewards to maintain source-traceable information flow across tool calls.<n>Experiments on two in-distribution (ID) and three out-of-distribution (OOD) benchmarks show that ToolLIFT consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-08-04T11:02:45Z)
• Beyond the Eye: Efficient Multimodal Reasoning via Self-Regulated Implicit Visual Tools [65.7098588070281]
  We propose Beyond the Eye (BEE), a novel implicit visual tool paradigm centered on self-regulated capability.<n>BEE incorporates visual tool invocation behaviors into the training objective and encourages the model to develop a self-regulated invocation mechanism.<n>BEE achieves state-of-the-art performance in fine-grained visual perception while remaining competitive in general reasoning tasks.
  arXiv  Detail & Related papers  (2026-07-13T05:32:00Z)
• From Intuition to Investigation: A Tool-Augmented Reasoning MLLM Framework for Generalizable Face Anti-Spoofing [47.26568822352609]
  We propose the Tool-Augmented Reasoning FAS (TAR-FAS) framework, which reformulates the Face Anti-Spoofing task as a Chain-of-Thought with Visual Tools (CoT-VT) paradigm.<n>We show that TAR-FAS achieves SOTA performance while providing fine-grained visual investigation for trustworthy spoof detection.
  arXiv  Detail & Related papers  (2026-03-01T10:28:35Z)
• ForgeryVCR: Visual-Centric Reasoning via Efficient Forensic Tools in MLLMs for Image Forgery Detection and Localization [62.03035862528452]
  ForgeryVCR is a framework that materializes imperceptible traces into explicit visual intermediates via Visual-Centric Reasoning.<n>ForgeryVCR achieves state-of-the-art (SOTA) performance in both detection and localization tasks.
  arXiv  Detail & Related papers  (2026-02-15T11:14:47Z)
• Evolving from Tool User to Creator via Training-Free Experience Reuse in Multimodal Reasoning [16.12114923351562]
  We propose a training-free framework that transforms agents from tool users to tool creators.<n>This approach harvests reasoning experiences and distills them into reusable assets.<n>We also introduce a memory consolidation mechanism to maintain the tool library.
  arXiv  Detail & Related papers  (2026-02-02T11:37:45Z)
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments [70.42705564227548]
  We propose an automated environment construction pipeline for large language models (LLMs)<n>This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools.<n>We also introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution.
  arXiv  Detail & Related papers  (2025-08-12T09:45:19Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use.<n>MeCo quantifies metacognitive scores by capturing high-level cognitive signals in the representation space.<n>MeCo is fine-tuning-free and incurs minimal cost.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• LLM With Tools: A Survey [0.0]
  This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools. We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans. Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes.
  arXiv  Detail & Related papers  (2024-09-24T14:08:11Z)
• Structural Embeddings of Tools for Large Language Models [0.0]
  It is evident that the current state of Large Language Models (LLMs) necessitates the incorporation of external tools. The ontological nature of tool utilization for a specific task can be well formulated with a Directed Acyclic Graph (DAG) We propose an exemplary framework to guide the orchestration of exponentially increasing numbers of external tools with LLMs.
  arXiv  Detail & Related papers  (2023-08-01T10:46:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.