When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models

• URL: http://arxiv.org/abs/2603.03989v1
• Date: Wed, 04 Mar 2026 12:33:36 GMT
• Title: When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models
• Abstract summary: Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of this behavior.<n>We introduce a representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images.
• Score: 0.5461938536945722
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When visual evidence is ambiguous, vision models must decide whether to interpret face-like patterns as meaningful. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of this behavior. We introduce a representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images. Under a unified protocol, we evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), general object detection (YOLOv8), and face detection (RetinaFace). Our analysis reveals three mechanisms of interpretation under ambiguity. VLMs exhibit semantic overactivation, systematically pulling ambiguous non-human regions toward the Human concept, with LLaVA-1.5-7B producing the strongest and most confident over-calls, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. These results show that behavior under ambiguity is governed more by representational choices than score thresholds, and that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. Pareidolia therefore provides a compact diagnostic and a source of ambiguity-aware hard negatives for probing and improving the semantic robustness of vision-language systems. Code will be released upon publication.

Related papers

• Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models [35.534692995586035]
  ASD diagnosis requires behavioral and developmental evidence, not static facial photographs.<n>We introduce PARITY, a synthetic, demographically balanced set of identity-controlled neutral/expression portrait pairs with neutral-neutral controls.<n>We find a clear split between refusal-first models and speculative models; in the latter, certain expressions disproportionately trigger harmful selections.
  arXiv  Detail & Related papers  (2026-09-02T11:52:02Z)
• Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs [49.743008490747094]
  Vision-Language Models (VLMs) remain prone to hallucinations, producing content that is inconsistent with or unsupported by the input image.<n>Under a unified head-level view, we find that hallucination-induced changes manifest as localized deviations from each head's faithful contextual behavior.<n> Detailed analysis reveals that these deviations are systematically organized across attention heads, contextual sources, and deviation directions, and that the resulting signal is linearly readable once head identity is preserved.
  arXiv  Detail & Related papers  (2026-07-31T13:33:51Z)
• Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity? [14.469911934779558]
  Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions.<n>We propose Visual Semantic Entropy (VSE), which perturbs only the image to probe nearby visual variations while keeping the text query fixed.<n>VSE effectively captures visual ambiguity, establishing a new state-of-the-art for VLM uncertainty estimation.
  arXiv  Detail & Related papers  (2026-06-30T09:35:20Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision? [10.315515647818009]
  Benchmark accuracy is often assumed to reflect grounded visual understanding in vision-language models.<n>We systematically investigate this mismatch in a set of open-source vision-language models.<n>We find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence.
  arXiv  Detail & Related papers  (2026-05-21T17:35:04Z)
• Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models [62.932580559941414]
  Vision-Language Models (VLMs) frequently "hallucinate" - generate plausible yet factually incorrect statements.<n>We propose a new paradigm for diagnosing hallucinations, recasting them from static output errors into dynamic pathologies of a model's computational cognition.
  arXiv  Detail & Related papers  (2026-03-16T17:20:38Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering [33.3588824921581]
  We introduce Vision-Language Introspection (VLI), a training-free inference framework that simulates a metacognitive self-correction process.<n>VLI achieves state-of-the-art performance on advanced models, reducing object hallucination rates by 12.67% on MMHal-Bench and improving accuracy by 5.8% on POPE.
  arXiv  Detail & Related papers  (2026-01-08T17:49:13Z)
• Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation [13.03365340564181]
  Hallucination is a major flaw in vision-language models and is particularly critical in sign language translation.<n>We propose a token-level reliability measure that quantifies how much the decoder uses visual information.<n>Our results show that reliability predicts hallucination rates, generalizes across datasets and architectures, and decreases under visual degradations.
  arXiv  Detail & Related papers  (2025-10-21T09:13:46Z)
• RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation [67.38036090822982]
  We propose RoboView-Bias, the first benchmark specifically designed to quantify visual bias in robotic manipulation.<n>We create 2,127 task instances that enable robust measurement of biases induced by individual visual factors and their interactions.<n>Our results highlight that systematic analysis of visual bias is a prerequisite for developing safe and reliable general-purpose embodied agents.
  arXiv  Detail & Related papers  (2025-09-26T13:53:25Z)
• Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models [24.363156120809546]
  We propose KIE-HVQA, the first benchmark dedicated to evaluating OCR hallucination in degraded document understanding.<n>This dataset includes test samples spanning identity cards and invoices, with simulated real-world degradations for OCR reliability.<n>Experiments on Qwen2.5-VL demonstrate that our 7B- parameter model achieves a 22% absolute improvement in hallucination-free accuracy over GPT-4o.
  arXiv  Detail & Related papers  (2025-06-25T06:44:07Z)
• LLMs Are Not Yet Ready for Deepfake Image Detection [8.364956401923108]
  Vision-language models (VLMs) have emerged as promising tools across various domains.<n>This study focuses on three primary deepfake types: faceswap, reenactment, and synthetic generation.<n>Our analysis indicates that while VLMs can produce coherent explanations and detect surface-level anomalies, they are not yet dependable as standalone detection systems.
  arXiv  Detail & Related papers  (2025-06-12T08:27:24Z)
• Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models [57.58426038241812]
  Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks.<n>These models still suffer from hallucinations when required to implicitly recognize or infer diverse visual entities from images.<n>We propose a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks.
  arXiv  Detail & Related papers  (2024-12-29T23:56:01Z)
• Unsupervised Model Diagnosis [49.36194740479798]
  This paper proposes Unsupervised Model Diagnosis (UMO) to produce semantic counterfactual explanations without any user guidance. Our approach identifies and visualizes changes in semantics, and then matches these changes to attributes from wide-ranging text sources.
  arXiv  Detail & Related papers  (2024-10-08T17:59:03Z)
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models [103.64435911083432]
  We present a novel contrastive learning strategy called it Proactive Pseudo-Intervention (PPI) PPI leverages proactive interventions to guard against image features with no causal relevance. We also devise a novel causally informed salience mapping module to identify key image pixels to intervene, and show it greatly facilitates model interpretability.
  arXiv  Detail & Related papers  (2020-12-06T20:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.