SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions

• URL: http://arxiv.org/abs/2603.23118v1
• Date: Tue, 24 Mar 2026 12:15:31 GMT
• Title: SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions
• Abstract summary: Multimodal Large Language Models (MLLMs) are highly vulnerable to hidden-pattern visual illusions.<n>This deficiency highlights a perceptual misalignment between current MLLMs and humans, and also introduces potential safety concerns.<n>We propose the Strategy of Multi-Scale Perception (SMSP), a plug-and-play framework that aligns with human visual perceptual strategies.
• Score: 46.29380423673203
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent works have shown that Multimodal Large Language Models (MLLMs) are highly vulnerable to hidden-pattern visual illusions, where the hidden content is imperceptible to models but obvious to humans. This deficiency highlights a perceptual misalignment between current MLLMs and humans, and also introduces potential safety concerns. To systematically investigate this failure, we introduce IlluChar, a comprehensive and challenging illusion dataset, and uncover a key underlying mechanism for the models' failure: high-frequency attention bias, where the models are easily distracted by high-frequency background textures in illusion images, causing them to overlook hidden patterns. To address the issue, we propose the Strategy of Multi-Scale Perception (SMSP), a plug-and-play framework that aligns with human visual perceptual strategies. By suppressing distracting high-frequency backgrounds, SMSP generates images closer to human perception. Our experiments demonstrate that SMSP significantly improves the performance of all evaluated MLLMs on illusion images, for instance, increasing the accuracy of Qwen3-VL-8B-Instruct from 13.0% to 84.0%. Our work provides novel insights into MLLMs' visual perception, and offers a practical and robust solution to enhance it. Our code is publicly available at https://github.com/Tujz2023/SMSP.

Related papers

• Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models [58.91911788912665]
  We propose Latent Visual Reconstruction (LaVer), a novel training framework that facilitates MLLMs in learning more discrimi visual representations.<n>Our method offers direct visual activation to MLLMs, which exhibit increased visual attention allocation, indicating enhanced utilization of visual information.
  arXiv  Detail & Related papers  (2025-12-06T04:20:13Z)
• One More Glance with Sharp Eyes: Rethinking Lightweight Captioning as a Practical Visual Specialist [58.89538703878721]
  We build lightweight captioning models using a 125M- parameter language model, 56 times smaller than LLaMA-7B.<n>We evaluate their performance on single-sentence but on detailed captioning tasks.<n>We develop a novel captioning framework, Sharp-Eyed Refinement, which enhances caption quality by refining coarse descriptions into more precise captions.
  arXiv  Detail & Related papers  (2025-08-29T09:29:27Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking [31.356290235986332]
  Vision-language models (VLMs) excel in semantic tasks but falter at a core human capability.<n>We introduce HC-Bench, a benchmark of 112 images with hidden text, objects, and illusions.<n>We propose SemVink (Semantic Visual Thinking), which unlocks >99% accuracy by eliminating redundant visual noise.
  arXiv  Detail & Related papers  (2025-06-03T12:33:47Z)
• MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction [6.416957959150438]
  Hallucinations hinder the application of Large Vision-Language Models (LVLMs) in domains that require high reliability.<n>We propose MINT, a training-free decoding strategy, MItigating hallucinations via tokeN reducTion.<n>Our approach achieves a 4% improvement in mitigating hallucinations caused by distracted perception compared to original models.
  arXiv  Detail & Related papers  (2025-02-02T08:34:57Z)
• Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink [22.781487074263957]
  Multi-modal Large Language Models (MLLMs) are revolutionizing visual-language applications.<n>Yet, these models are often plagued by the hallucination problem, which involves generating inaccurate objects, attributes, and relationships that do not match the visual content.<n>We propose a novel hallucination attack against MLLMs that exploits attention sink behaviors to trigger hallucinated content with minimal image-text relevance.
  arXiv  Detail & Related papers  (2025-01-25T16:36:00Z)
• Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning [151.4060202671114]
  multimodal large language models (MLLMs) have shown unprecedented capabilities in advancing vision-language tasks.<n>This paper introduces a novel bottom-up reasoning framework to address hallucinations in MLLMs.<n>Our framework systematically addresses potential issues in both visual and textual inputs by verifying and integrating perception-level information with cognition-level commonsense knowledge.
  arXiv  Detail & Related papers  (2024-12-15T09:10:46Z)
• Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance [67.26434607115392]
  Large vision-language models (LVLMs) have achieved impressive results in various vision-language tasks. LVLMs suffer from hallucinations caused by language bias, leading to diminished focus on images and ineffective visual comprehension. We propose LACING to address the language bias of LVLMs with muLtimodal duAl-attention meChanIsm (MDA) aNd soft-image Guidance (IFG)
  arXiv  Detail & Related papers  (2024-11-21T16:33:30Z)
• From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models [15.401221354325672]
  Hallucinations in large vision models (LVLMs) are a significant challenge, i.e., generating objects that are not presented in the visual input.<n>Recent studies often attribute hallucinations to a lack of understanding of visual input, yet ignore a more fundamental issue: the model's inability to extract or decouple visual features.<n>In this paper, we revisit the hallucinations in LVLMs from an architectural perspective, investigating whether the primary cause lies in the visual encoder (feature extraction) or the modal alignment module (feature decoupling)
  arXiv  Detail & Related papers  (2024-10-09T11:46:32Z)
• Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models [121.83413400686139]
  This paper proposes to improve the visual perception ability of MLLMs through a mixture-of-experts knowledge enhancement mechanism. We introduce a novel method that incorporates multi-task encoders and visual tools into the existing MLLMs training and inference pipeline.
  arXiv  Detail & Related papers  (2024-01-06T02:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.