Diagnosing Visual Ignorance in Vision-Language Models

• URL: http://arxiv.org/abs/2606.06890v1
• Date: Fri, 05 Jun 2026 04:16:01 GMT
• Title: Diagnosing Visual Ignorance in Vision-Language Models
• Abstract summary: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
• Score: 29.2851901986069
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence. While this behavior is widely observed, its internal mechanisms and its impact on benchmark evaluation remain insufficiently understood. In this work, we study language-prior reliance from both mechanistic and behavioral perspectives. Internally, we combine counterfactual layer replacement with supervised layer-wise MLP probing to trace how ground-truth visual semantics and language-prior semantics compete across the language decoder. Our analysis reveals a multi-stage bottleneck: intermediate layers often fail to effectively retrieve visual information, while later layers can further suppress surviving visual signals in favor of text-space biases. Externally, we introduce a progressive visual decay metric based on multi-step Gaussian blurring, which identifies instances whose answers remain invariant even as visual content is increasingly destroyed. Across twelve visual question-answering benchmarks and three representative VLMs, we find that a substantial fraction of examples remain answerable under severe or total visual obfuscation, indicating that current benchmarks can inadvertently reward visual ignorance. These findings demonstrate that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity. Finally, we outline critical pathways for future research, highlighting the necessity of designing training distributions and evaluation protocols built on structurally isolated or counterfactual data to enforce genuine cross-modal grounding.

Related papers

• What Do Hallucinations Reveal About Multimodal Reasoning? Diagnosing Visual Grounding Failures via Contrastive Decoding Probes [1.9916820095972823]
  We introduce SAFE, a training-free decoding framework that contrasts visually-grounded and vision-ablated generation paths.<n>This signal serves dual roles: as a practical proxy for detecting visually-ungrounded tokens, and as the basis for decoding-time penalties.<n>We argue that designing contrastive probes exemplifies a broader mission: using models as instruments for scientific understanding.
  arXiv  Detail & Related papers  (2026-09-15T05:07:01Z)
• Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models [14.5102059776721]
  Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models.<n>However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge.<n>We explore these failures using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity.
  arXiv  Detail & Related papers  (2026-07-28T22:52:34Z)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis [63.49939245944781]
  We introduce FG-BMK, a comprehensive fine-grained evaluation benchmark containing 1.01 million questions and 0.28 million images.<n>We find that current LVLMs remain inadequate fine-grained recognizers, with failures arising from intertwined bottlenecks in visual representations, semantic grounding, modality alignment, and category-level knowledge.
  arXiv  Detail & Related papers  (2026-06-17T13:23:26Z)
• Unveiling the Visual Counting Bottleneck in Vision-Language Models [49.591496870141846]
  This work deconstructs visual counting into three cognitive stages: visual individuation, magnitude awareness, and symbolic mapping.<n>Using synthetic Go boards and linear probes, we demonstrate that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime.<n>We pinpoint the collapse to the symbolic mapping stage, where the model fails to project valid visual magnitudes onto symbolic tokens.
  arXiv  Detail & Related papers  (2026-05-28T16:20:29Z)
• Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision? [10.315515647818009]
  Benchmark accuracy is often assumed to reflect grounded visual understanding in vision-language models.<n>We systematically investigate this mismatch in a set of open-source vision-language models.<n>We find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence.
  arXiv  Detail & Related papers  (2026-05-21T17:35:04Z)
• Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation [31.028607494171336]
  Vision-Language Models (VLMs) are frequently undermined by object hallucination.<n>We introduce Positive-and-Negative Decoding (PND), a training-free inference framework.<n>PND intervenes directly in the decoding process to enforce visual fidelity.
  arXiv  Detail & Related papers  (2026-04-27T12:23:00Z)
• Counting to Four is still a Chore for VLMs [3.017198998175514]
  Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks.<n>They still fail on simple grounding skills such as object counting.<n>We present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis.
  arXiv  Detail & Related papers  (2026-04-11T05:23:19Z)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models [84.94288033791346]
  We conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs.<n>We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation.<n>We propose Predictive Regularization to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations.
  arXiv  Detail & Related papers  (2026-03-21T13:10:37Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy [59.44168425139687]
  BayesVLA is a Bayesian factorization that decomposes the policy into a visual-action prior, supporting seeing-to-act, and a language-conditioned likelihood, enabling prompt-to-specify.<n>Experiments show superior generalization to unseen instructions, objects, and environments compared to existing methods.
  arXiv  Detail & Related papers  (2025-12-12T01:59:23Z)
• Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding [12.42420657573375]
  We present the first systematic analysis of language prior in large vision-language models (LVLMs)<n>Our analysis reveals a universal phenomenon: each model exhibits a Visual Integration Point (VIP), a critical layer at which visual information begins to meaningfully reshape hidden representations and influence decoding.<n>Across 54 model-dataset combinations spanning 9 contemporary LVLMs and 6 benchmarks, we demonstrate that VIP consistently emerges, and that TVI reliably predicts the strength of language prior.
  arXiv  Detail & Related papers  (2025-09-27T02:12:05Z)
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models [57.43276586087863]
  Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs. Existing benchmarks are often limited in scope, focusing mainly on object hallucinations. We introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases.
  arXiv  Detail & Related papers  (2024-04-22T04:49:22Z)
• Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem [60.0878532426877]
  We propose a novel collaborative learning scheme from the viewpoint of visual perturbation calibration. Specifically, we devise a visual controller to construct two sorts of curated images with different perturbation extents. The experimental results on two diagnostic VQA-CP benchmark datasets evidently demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2022-07-24T23:50:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.