The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

• URL: http://arxiv.org/abs/2608.06270v1
• Date: Thu, 06 Aug 2026 17:01:08 GMT
• Title: The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
• Abstract summary: "thinking-with-images" operations often achieve only marginal or negative gains over direct inference at substantially higher token cost.<n>We ask whether the returned visual evidence causally affects the answer.<n>We use visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts.
• Score: 19.417504594148223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.

Related papers

• Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs [14.028133377242701]
  Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.<n>We introduce PriVE-Bench, a Prior-vs-Visual Evidence Benchmark that uses paired original and counterfactual images to distinguish visually grounded answers from prior-consistent errors.<n>We also introduce PriVE-Tools, a controlled agentic-vision-inspired extension that evaluates whether tool-derived visual evidence improves grounding under the same counterfactual conflicts.
  arXiv  Detail & Related papers  (2026-07-14T16:45:02Z)
• Interaction Scaling: Grounding the Third Axis of Test-Time Compute [0.33451037881913753]
  We argue that a single variable governs this third axis, grounding, and that it must hold on both sides of the loop.<n>On hard coding tasks at a fixed token budget, reasoning-only and best-of-N sampling both plateau.<n>Our proposer-reviewer harness reaches a perfect 100% pass rate with no run-to-run variance.
  arXiv  Detail & Related papers  (2026-07-13T14:22:06Z)
• VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning? [79.44137632338062]
  When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence?<n>We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question pairs across cardinality, attribute, spatial, and logic tasks.<n>We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images.
  arXiv  Detail & Related papers  (2026-06-05T22:06:07Z)
• Unveiling the Visual Counting Bottleneck in Vision-Language Models [49.591496870141846]
  This work deconstructs visual counting into three cognitive stages: visual individuation, magnitude awareness, and symbolic mapping.<n>Using synthetic Go boards and linear probes, we demonstrate that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime.<n>We pinpoint the collapse to the symbolic mapping stage, where the model fails to project valid visual magnitudes onto symbolic tokens.
  arXiv  Detail & Related papers  (2026-05-28T16:20:29Z)
• Pando: Do Interpretability Methods Work When Models Won't Explain Themselves? [53.07826484214082]
  We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis.<n>Pando predicts held-out model decisions from $10$ labeled query-response pairs.<n>When explanations are faithful, black-box elicitation matches or exceeds all white-box methods.
  arXiv  Detail & Related papers  (2026-04-13T06:42:24Z)
• Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts [7.071615105094556]
  We show that visual attributes can be linearly decodable from early layers.<n>We show that training-free activation in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups.
  arXiv  Detail & Related papers  (2026-04-10T14:36:07Z)
• CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation [0.996038809132083]
  We introduce CUE-R, a framework for measuring per-evidence-item operational utility in single-shot RAG.<n>We show that CUE-R perturbs individual evidence items via REMOVE, REPLACE, and DUPLICATE operators, then measures changes along three utility axes.<n>Our results suggest that answer-only evaluation misses important evidence effects and that intervention-based utility analysis is a practical complement for RAG evaluation.
  arXiv  Detail & Related papers  (2026-04-07T06:05:08Z)
• To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs [1.9349092719498848]
  Tri-Layer Diagnostic Framework disentangles hallucination sources via three metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score.<n>Our taxonomy reveals that 69.6% of samples exhibit Visual Sycophancy--models detect visual anomalies but hallucinate to satisfy user expectations--while zero samples show Robust Refusal.
  arXiv  Detail & Related papers  (2026-03-19T00:15:05Z)
• Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection [49.26064449816502]
  We propose a Gradient-based Influence-Aware Constrained Decoding (GACD) method to address text-visual bias and co-occurrence bias.<n>GACD effectively reduces hallucinations and improves the visual grounding of MLLM outputs.
  arXiv  Detail & Related papers  (2025-09-03T08:13:52Z)
• Regularized Contrastive Partial Multi-view Outlier Detection [76.77036536484114]
  We propose a novel method named Regularized Contrastive Partial Multi-view Outlier Detection (RCPMOD) In this framework, we utilize contrastive learning to learn view-consistent information and distinguish outliers by the degree of consistency. Experimental results on four benchmark datasets demonstrate that our proposed approach could outperform state-of-the-art competitors.
  arXiv  Detail & Related papers  (2024-08-02T14:34:27Z)
• Causal Transportability for Visual Recognition [70.13627281087325]
  We show that standard classifiers fail because the association between images and labels is not transportable across settings. We then show that the causal effect, which severs all sources of confounding, remains invariant across domains. This motivates us to develop an algorithm to estimate the causal effect for image classification.
  arXiv  Detail & Related papers  (2022-04-26T15:02:11Z)
• Can contrastive learning avoid shortcut solutions? [88.249082564465]
  implicit feature modification (IFM) is a method for altering positive and negative samples in order to guide contrastive models towards capturing a wider variety of predictive features. IFM reduces feature suppression, and as a result improves performance on vision and medical imaging tasks.
  arXiv  Detail & Related papers  (2021-06-21T16:22:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.