Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning

• URL: http://arxiv.org/abs/2608.09682v1
• Date: Mon, 10 Aug 2026 14:52:10 GMT
• Title: Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning
• Abstract summary: Tool-augmented vision-language models increasingly "think with images"<n>Recent work using blind tests, gain decompositions, and attention analyses has shown that returned images contribute little.<n>We hypothesize that the load-bearing signal is the structured text emitted before any returned pixel arrives.
• Score: 75.00741348880044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tool-augmented vision-language models increasingly "think with images": they call crop, zoom, or code tools and reason over the returned pixels. However, recent work using blind tests, gain decompositions, and attention analyses has shown that returned images contribute little, raising the question: if pixels do not carry the gain, what does? We hypothesize that the load-bearing signal is the structured text emitted before any returned pixel arrives: tool name, coordinates, target description, and intent. This textual scaffold encodes where to look and what to find. We introduce TextCall (call-but-no-return) to test this: it keeps the scaffold but replaces returned images with the text placeholder [Image output skipped]. Three studies support the hypothesis. (i) Non-necessity of returned pixels: across LoRA, full fine-tuning, and RL, TextCall matches or exceeds full thinking-with-images; under RL it preserves tool use at the reported checkpoint, avoiding the failure mode where, under matched settings, seeing the returned image causes the model to stop calling tools and answer directly. (ii) Sufficiency of the scaffold: on matched training queries, scaffold-only input yields equivalent accuracy to returned-image input. (iii) Component specificity: decomposing the scaffold into reasoning text and spatial code shows both components contribute, with the dominant one varying by task. Together these results support the Tool-Call Scaffold Hypothesis: in current thinking-with-images distributions, the active signal is the structured text emitted at tool-call time; the returned image is a redundant carrier. TextCall preserves accuracy while reducing latency by 29-46% and eliminating tool-execution API calls. Our claims hold for current thinking-with-images benchmarks; constructing tasks where pixels are genuinely load-bearing remains an open direction.

Related papers

• FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification [41.86420087238674]
  Recent studies have revealed that agentic vision-language models often use tools unfaithfully.<n>We introduce FaithEyes, a multi-agent self-judging framework.<n>FaithEyes attains competitive or superior accuracy across visual perception and reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-07-30T13:58:08Z)
• PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation [70.26672362704758]
  We introduce PixelRAG, a new retrieval-augmented method that represents websites in their native visual form.<n> PixelRAG is the first pipeline to operate over a full Wikipedia corpus in this form.<n>It consistently outperforms both no-retrieval and text-based RAG baselines.
  arXiv  Detail & Related papers  (2026-06-01T23:20:51Z)
• Exploring Fine-Grained Image-Text Alignment for Referring Remote Sensing Image Segmentation [27.13782704236074]
  We propose a new referring remote sensing image segmentation method to fully exploit the visual and linguistic representations.<n>The proposed fine-grained image-text alignment module (FIAM) would simultaneously leverage the features of the input image and the corresponding texts.<n>We evaluate the effectiveness of the proposed method on two public referring remote sensing datasets including RefSegRS and RRSIS-D.
  arXiv  Detail & Related papers  (2024-09-20T16:45:32Z)
• Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generation [72.6168579583414]
  CompAgent is a training-free approach for compositional text-to-image generation with a large language model (LLM) agent as its core. Our approach achieves more than 10% improvement on T2I-CompBench, a comprehensive benchmark for open-world compositional T2I generation.
  arXiv  Detail & Related papers  (2024-01-28T16:18:39Z)
• TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification [59.779532652634295]
  We propose an embarrassingly simple approach to better align image and text features with no need of additional data formats other than image-text pairs. We parse objects and attributes from the description, which are highly likely to exist in the image. Experiments substantiate the average 5.2% improvement of our framework over existing alternatives.
  arXiv  Detail & Related papers  (2023-12-21T18:59:06Z)
• Decompose Semantic Shifts for Composed Image Retrieval [38.262678009072154]
  Composed image retrieval is a type of image retrieval task where the user provides a reference image as a starting point and specifies a text on how to shift from the starting point to the desired target image. We propose a Semantic Shift network (SSN) that explicitly decomposes the semantic shifts into two steps: from the reference image to the visual prototype and from the visual prototype to the target image. The proposed SSN demonstrates a significant improvement of 5.42% and 1.37% on the CIRR and FashionIQ datasets, respectively, and establishes a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-09-18T07:21:30Z)
• Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback [20.78162037954646]
  We introduce a decompositional approach towards evaluation and improvement of text-to-image alignment. Human user studies indicate that the proposed approach surpasses previous state-of-the-art by 8.7% in overall text-to-image alignment accuracy.
  arXiv  Detail & Related papers  (2023-07-10T17:54:57Z)
• Re-Imagen: Retrieval-Augmented Text-to-Image Generator [58.60472701831404]
  Retrieval-Augmented Text-to-Image Generator (Re-Imagen) Retrieval-Augmented Text-to-Image Generator (Re-Imagen)
  arXiv  Detail & Related papers  (2022-09-29T00:57:28Z)
• Detector-Free Weakly Supervised Grounding by Separation [76.65699170882036]
  Weakly Supervised phrase-Grounding (WSG) deals with the task of using data to learn to localize arbitrary text phrases in images. We propose Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector. We demonstrate a significant accuracy improvement, of up to $8.5%$ over previous DF-WSG SotA.
  arXiv  Detail & Related papers  (2021-04-20T08:27:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.