Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions

• URL: http://arxiv.org/abs/2603.29428v1
• Date: Tue, 31 Mar 2026 08:35:10 GMT
• Title: Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions
• Abstract summary: Vision-language models (VLMs) exhibit a systematic bias when confronted with classic optical illusions.<n>We present a tool-guided inference framework that addresses this failure mode without any model training.<n>We report three empirical observations that we believe warrant additional investigation.
• Score: 2.9521041993295607
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) exhibit a systematic bias when confronted with classic optical illusions: they overwhelmingly predict the illusion as "real" regardless of whether the image has been counterfactually modified. We present a tool-guided inference framework for the DataCV 2026 Challenge (Tasks I and II) that addresses this failure mode without any model training. An off-the-shelf vision-language model is given access to a small set of generic image manipulation tools: line drawing, region cropping, side-by-side comparison, and channel isolation, together with an illusion-type-routing system prompt that prescribes which tools to invoke for each perceptual question category. Critically, every tool call produces a new, immutable image resource appended to a persistent registry, so the model can reference and compose any prior annotated view throughout its reasoning chain. Rather than hard-coding illusion-specific modules, this generic-tool-plus-routing design yields strong cross-structural generalization: performance remained consistent from the validation set to a test set containing structurally unfamiliar illusion variants (e.g., Mach Bands rotated from vertical to horizontal stacking). We further report three empirical observations that we believe warrant additional investigation: (i) a strong positive-detection bias likely rooted in imbalanced illusion training data, (ii) a striking dissociation between pixel-accurate spatial reasoning and logical inference over self-generated annotations, and (iii) pronounced sensitivity to image compression artifacts that compounds false positives.

Related papers

• Improving Reasoning in Vision-Language Models via Perception Verified Self-Training [3.443084677278651]
  We propose a perception-verified self-training framework that enforces visually grounded reasoning.<n>First, our method employs a CoT template that disentangles perception from reasoning, enabling independent verification of visual understanding.<n>Using caption verification together with answer correctness, we partition the data into three subsets: easy (correct caption and conclusion), medium (correct caption but incorrect conclusion), and hard (incorrect caption)
  arXiv  Detail & Related papers  (2026-06-20T17:33:07Z)
• Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning [14.700305370711973]
  We propose a training-free, data-centric framework designed to fortify visual grounding in frozen Vision-Language Models.<n>Our framework was evaluated on the DataCV 2026 Challenge (Task I: Classic Illusion Understanding), where it ranked 2nd place overall.<n>Our success underscores the potential of structured qualitative grounding as a robust paradigm for developing next-generation, illusion-resistant vision-language systems.
  arXiv  Detail & Related papers  (2026-04-29T03:12:29Z)
• Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigation [48.06125338526647]
  Large vision-language models (LVLMs) are prone to hallucinations in multi-image tasks.<n>We propose a structured hallucination mitigation framework involving Cross-Image Attention calibration and Preference Learning.
  arXiv  Detail & Related papers  (2026-03-07T05:57:27Z)
• VAAS: Vision-Attention Anomaly Scoring for Image Manipulation Detection in Digital Forensics [0.0]
  Recent advances in AI-driven image generation have introduced new challenges for verifying the authenticity of digital evidence in forensic investigations.<n>Modern generative models can produce visually consistent forgeries that evade traditional detectors based on pixel or compression artefacts.<n>This paper introduces Vision-Attention Anomaly Scoring (VAAS), a novel dual-module framework that integrates global attention-based anomaly estimation.
  arXiv  Detail & Related papers  (2025-12-17T15:05:40Z)
• Leveraging Hierarchical Image-Text Misalignment for Universal Fake Image Detection [58.927873049646024]
  We show that fake images cannot be properly aligned with corresponding captions compared to real images.<n>We propose a simple yet effective ITEM by leveraging the image-text misalignment in a joint visual-language space as discriminative clues.
  arXiv  Detail & Related papers  (2025-11-01T06:51:14Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• RT-VLM: Re-Thinking Vision Language Model with 4-Clues for Real-World Object Recognition Robustness [2.9979091009694088]
  Real world deployments often expose modern object recognition models to domain shifts that precipitate a severe drop in accuracy.<n>To mitigate this degradation, we introduce the Re-Thinking Vision Language Model (RT-VLM) framework.
  arXiv  Detail & Related papers  (2025-09-01T02:13:00Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpoints [42.64942578228025]
  We propose a novel method called ViewFool to find adversarial viewpoints that mislead visual recognition models. By encoding real-world objects as neural radiance fields (NeRF), ViewFool characterizes a distribution of diverse adversarial viewpoints.
  arXiv  Detail & Related papers  (2022-10-08T03:06:49Z)
• Warp Consistency for Unsupervised Learning of Dense Correspondences [116.56251250853488]
  Key challenge in learning dense correspondences is lack of ground-truth matches for real image pairs. We propose Warp Consistency, an unsupervised learning objective for dense correspondence regression. Our approach sets a new state-of-the-art on several challenging benchmarks, including MegaDepth, RobotCar and TSS.
  arXiv  Detail & Related papers  (2021-04-07T17:58:22Z)
• SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses [82.56853587380168]
  We propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of the same scene from different lens should be the same. We leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters. Our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-30T08:23:25Z)
• Unsupervised Landmark Learning from Unpaired Data [117.81440795184587]
  Recent attempts for unsupervised landmark learning leverage synthesized image pairs that are similar in appearance but different in poses. We propose a cross-image cycle consistency framework which applies the swapping-reconstruction strategy twice to obtain the final supervision. Our proposed framework is shown to outperform strong baselines by a large margin.
  arXiv  Detail & Related papers  (2020-06-29T13:57:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.