Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?

• URL: http://arxiv.org/abs/2606.31407v1
• Date: Tue, 30 Jun 2026 09:35:20 GMT
• Title: Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?
• Abstract summary: Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions.<n>We propose Visual Semantic Entropy (VSE), which perturbs only the image to probe nearby visual variations while keeping the text query fixed.<n>VSE effectively captures visual ambiguity, establishing a new state-of-the-art for VLM uncertainty estimation.
• Score: 14.469911934779558
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions. Common entropy-based methods, such as Semantic Entropy (SE), rely on output diversity. Yet our analysis shows that overconfident visual embeddings suppress output diversity under stochastic decoding, causing SE to underestimate uncertainty in such cases. Recent methods instead probe output diversity through input perturbations, including textual paraphrasing or joint text-image perturbations, and show improved performance. We study these approaches and reveals that the resulting variability is often dominated by textual changes rather than visual evidence, causing uncertainty estimates to reflect prompt sensitivity rather than visual ambiguity. We therefore propose Visual Semantic Entropy (VSE), which perturbs only the image to probe nearby visual variations while keeping the text query fixed. VSE measures uncertainty by clustering generated answers into semantic prototypes and computing the mass-weighted dispersion among them. Extensive evaluation across five modern vision-language models and five diverse VQA benchmarks demonstrates that VSE effectively captures visual ambiguity, establishing a new state-of-the-art for VLM uncertainty estimation.

Related papers

• Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation [10.212763020484092]
  Uncertainty quantification (UQ) remains a critical challenge in Large Vision Language Models (LVLMs)<n>We propose Visual-Grounded Token UQ (VIG-TUQ), a training-free framework that explicitly incorporates visual grounding into uncertainty estimation.
  arXiv  Detail & Related papers  (2026-05-26T15:04:55Z)
• Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision? [10.315515647818009]
  Benchmark accuracy is often assumed to reflect grounded visual understanding in vision-language models.<n>We systematically investigate this mismatch in a set of open-source vision-language models.<n>We find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence.
  arXiv  Detail & Related papers  (2026-05-21T17:35:04Z)
• When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models [0.5461938536945722]
  Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of this behavior.<n>We introduce a representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images.
  arXiv  Detail & Related papers  (2026-03-04T12:33:36Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy [75.66913260900726]
  Reinforcement Learning with Verifiable Rewards has significantly advanced reasoning capabilities in Large Language Models.<n>Existing paradigms, driven by text-centric outcome rewards, encourage models to bypass visual perception.<n>We propose textbfThinking with Deltas, a framework driven by a textbfDifferential Visual Reasoning Policy.
  arXiv  Detail & Related papers  (2026-01-11T08:25:34Z)
• Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models [93.46875303598577]
  Vision-language models (VLMs) have advanced rapidly in processing multimodal information, but their ability to reconcile conflicting signals remains underexplored.<n>This work investigates how VLMs process ASCII art, a unique medium where textual elements collectively form visual patterns, potentially creating semantic-visual conflicts.
  arXiv  Detail & Related papers  (2025-04-02T10:47:07Z)
• From Uncertainty to Trust: Enhancing Reliability in Vision-Language Models with Uncertainty-Guided Dropout Decoding [6.115580421973011]
  Large vision-language models (LVLMs) demonstrate remarkable capabilities in multimodal tasks but are prone to misinterpreting visual inputs, often resulting in hallucinations and unreliable outputs.<n>We propose Dropout Decoding, a novel inference-time approach that quantifies the uncertainty of visual tokens and selectively masks uncertain tokens to improve decoding.<n> Evaluations on benchmarks including CHAIR, THRONE, and MMBench demonstrate that Dropout Decoding significantly reduces object hallucinations (OH) and enhances both reliability and quality of LVLM outputs.
  arXiv  Detail & Related papers  (2024-12-09T13:21:07Z)
• Beyond Coarse-Grained Matching in Video-Text Retrieval [50.799697216533914]
  We introduce a new approach for fine-grained evaluation. Our approach can be applied to existing datasets by automatically generating hard negative test captions. Experiments on our fine-grained evaluations demonstrate that this approach enhances a model's ability to understand fine-grained differences.
  arXiv  Detail & Related papers  (2024-10-16T09:42:29Z)
• Uncertainty in latent representations of variational autoencoders optimized for visual tasks [3.9504737666460037]
  We investigate the properties of inference in Variational Autoencoders (VAEs)<n>We draw inspiration from classical computer vision to introduce an inductive bias into the VAE.<n>We find that restored inference capabilities are delivered by developing a motif in the inference network.
  arXiv  Detail & Related papers  (2024-04-23T16:26:29Z)
• Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem [60.0878532426877]
  We propose a novel collaborative learning scheme from the viewpoint of visual perturbation calibration. Specifically, we devise a visual controller to construct two sorts of curated images with different perturbation extents. The experimental results on two diagnostic VQA-CP benchmark datasets evidently demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2022-07-24T23:50:52Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.