Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions

• URL: http://arxiv.org/abs/2601.22150v1
• Date: Thu, 29 Jan 2026 18:59:24 GMT
• Title: Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions
• Abstract summary: Large Vision-Language Models (VLMs) often answer classic visual illusions "correctly" on original images, yet persist with the same responses when illusion factors are inverted.<n>This raises a fundamental question: do VLMs perceive visual changes or merely recall memorized patterns?<n>This paper introduces VI-Probe, a controllable visual-illusion framework with graded perturbations and matched visual controls.
• Score: 38.42074629749954
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (VLMs) often answer classic visual illusions "correctly" on original images, yet persist with the same responses when illusion factors are inverted, even though the visual change is obvious to humans. This raises a fundamental question: do VLMs perceive visual changes or merely recall memorized patterns? While several studies have noted this phenomenon, the underlying causes remain unclear. To move from observations to systematic understanding, this paper introduces VI-Probe, a controllable visual-illusion framework with graded perturbations and matched visual controls (without illusion inducer) that disentangles visually grounded perception from language-driven recall. Unlike prior work that focuses on averaged accuracy, we measure stability and sensitivity using Polarity-Flip Consistency, Template Fixation Index, and an illusion multiplier normalized against matched controls. Experiments across different families reveal that response persistence arises from heterogeneous causes rather than a single mechanism. For instance, GPT-5 exhibits memory override, Claude-Opus-4.1 shows perception-memory competition, while Qwen variants suggest visual-processing limits. Our findings challenge single-cause views and motivate probing-based evaluation that measures both knowledge and sensitivity to controlled visual change. Data and code are available at https://sites.google.com/view/vi-probe/.

Related papers

• Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination [39.713386491199884]
  Vision-Language Models (VLMs) often produce self-reflective statements like "let me check the figure again" during reasoning.<n>Do such statements trigger genuine visual re-examination, or are they merely learned textual patterns?<n>We investigate this via VisualSwap, an image-swap probing framework.
  arXiv  Detail & Related papers  (2026-05-15T11:31:14Z)
• Illusion-Aware Visual Preprocessing and Anti-Illusion Prompting for Classic Illusion Understanding in Vision-Language Models [11.342804429826936]
  Vision-Language Models (VLMs) exhibit systematic bias toward visual illusions, recalling memorized facts rather than perceiving actual visual differences.<n>This paper presents a training-free framework for the 5th DataCV Challenge Task 1 at CVPR 2026, addressing this perception-versus-memory conflict.<n>Our solution secured 2nd place in the challenge, only 0.47% behind the 1st-place solution.
  arXiv  Detail & Related papers  (2026-05-09T09:52:23Z)
• Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs [60.93949629734977]
  We propose Visual Contrastive Self-Taught Reasoner (VC-STaR) to mitigate hallucinations in model-generated rationales.<n>We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modal similarity, and generate rationales using VC-STaR.<n>Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also surpasses models finetuned on the SoTA visual reasoning datasets.
  arXiv  Detail & Related papers  (2026-03-03T03:18:31Z)
• Self-Rewarding Vision-Language Model via Reasoning Decomposition [49.784411666601905]
  Vision-Language Models (VLMs) often suffer from visual hallucinations, saying things that are not actually in the image, and language shortcuts.<n>We introduce Vision-SR1, a self-rewarding method that improves visual reasoning without relying on external visual supervisions.<n>Our experiments demonstrate that Vision-SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts.
  arXiv  Detail & Related papers  (2025-08-27T08:01:03Z)
• See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs [7.964168958699652]
  This paper presents ViHallu, a Vision-Centric Hallucination mitigation framework.<n>ViHallu introduces visual variation images with controllable visual alterations while maintaining the overall image structure.<n>Experiments show that ViHallu effectively enhances models' fine-grained visual understanding while significantly reducing hallucination tendencies.
  arXiv  Detail & Related papers  (2025-07-29T16:53:27Z)
• Do Large Vision-Language Models Distinguish between the Actual and Apparent Features of Illusions? [12.157632635072435]
  Humans are susceptible to optical illusions, which serve as valuable tools for investigating sensory and cognitive processes.<n>Research has begun exploring whether machines, such as large vision language models (LVLMs), exhibit similar susceptibilities to visual illusions.
  arXiv  Detail & Related papers  (2025-06-06T05:47:50Z)
• IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models [56.34742191010987]
  Current Visual Language Models (VLMs) show impressive image understanding but struggle with visual illusions.<n>We introduce IllusionBench, a comprehensive visual illusion dataset that encompasses classic cognitive illusions and real-world scene illusions.<n>We design trap illusions that resemble classical patterns but differ in reality, highlighting issues in SOTA models.
  arXiv  Detail & Related papers  (2025-01-01T14:10:25Z)
• The Art of Deception: Color Visual Illusions and Diffusion Models [55.830105086695]
  Recent studies have shown that artificial neural networks (ANNs) can also be deceived by visual illusions.<n>We show how visual illusions are encoded in diffusion models.<n>We also show how to generate new unseen visual illusions in realistic images using text-to-image diffusion models.
  arXiv  Detail & Related papers  (2024-12-13T13:07:08Z)
• Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs [52.497823009176074]
  Large Vision-Language Models (LVLMs) often produce responses that misalign with factual information, a phenomenon known as hallucinations.<n>We introduce Visual Description Grounded Decoding (VDGD), a training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs.
  arXiv  Detail & Related papers  (2024-05-24T16:21:59Z)
• Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination [14.25488878224697]
  We propose Pensieve, a training-free method that leverages the analogous visual hallucinations, which are induced by images sharing common semantic and appearance characteristics. Pensieve mitigates the effects of addressing errors from both the visual and textual branches by adaptively scaling the subtracted scores.
  arXiv  Detail & Related papers  (2024-03-21T13:49:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.