Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking

• URL: http://arxiv.org/abs/2605.10893v2
• Date: Fri, 15 May 2026 03:21:31 GMT
• Title: Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking
• Abstract summary: We introduce BICR (Blind-Image Contrastive Ranking), a model-agnostic confidence estimation framework.<n>A lightweight probe is trained on the real-image hidden state and regularized by a ranking loss that penalizes higher confidence on the blacked-out view.<n> evaluated across five modern LVLMs and seven baselines on a benchmark covering visual question answering, object detection, medical imaging, and financial document understanding.
• Score: 7.845652284569666
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large vision-language models suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven entirely by language priors, with the image contributing nothing to the prediction. Existing confidence estimation methods cannot detect this, as they observe model behavior under normal inference with no mechanism to determine whether a prediction was shaped by the image or by text alone. We introduce BICR (Blind-Image Contrastive Ranking), a model-agnostic confidence estimation framework that makes this contrast explicit during training by extracting hidden states from a frozen LVLM twice: once with the real image-question pair, and once with the image blacked out while the question is held fixed. A lightweight probe is trained on the real-image hidden state and regularized by a ranking loss that penalizes higher confidence on the blacked-out view, teaching it to treat visual grounding as a signal of reliability at zero additional inference cost. Evaluated across five modern LVLMs and seven baselines on a benchmark covering visual question answering, object hallucination detection, medical imaging, and financial document understanding, BICR achieves the best cross-LVLM average on both calibration and discrimination simultaneously, with statistically significant discrimination gains robust to cluster-aware analysis at 4-18x fewer parameters than the strongest probing baseline.

Related papers

• Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning [66.13367818892503]
  We propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation.<n> TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants.<n>It combines image and question dependence with confidence to construct a response-level reward that favors jointly grounded responses.
  arXiv  Detail & Related papers  (2026-09-09T22:54:15Z)
• Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Models [14.059001230269132]
  Recent methods optimize verbalized confidence together with answer generation.<n> Verbalized confidence also provides no explicit assessment of visual support.<n>We propose textbfDualRead to separate capability learning from confidence estimation.
  arXiv  Detail & Related papers  (2026-09-06T06:34:01Z)
• Reliability Challenges in Diffusion Vision-Language Models [2.6281187565150526]
  Diffusion-based Large Vision-Language Models (dLVLMs) have emerged as a compelling alternative to autoregressive (AR) LVLMs.<n>We present the first systematic reliability evaluation of hallucination and bias in dLVLMs.
  arXiv  Detail & Related papers  (2026-09-01T14:38:50Z)
• Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models [4.0196226339562555]
  A vision-language model can answer a question about a medical image fluently and confidently while barely using the image.<n>We evaluate seven confidence estimators across five open-weight LVLMs and three medical visual-question-answering datasets.
  arXiv  Detail & Related papers  (2026-06-14T16:39:17Z)
• Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy [27.323597665236644]
  Large Vision-Language Models (LVLMs) show strong visualization interpretation.<n>It is unclear whether their responses reflect genuine reasoning over visual evidence or factual priors learned during training.<n>We present a framework that isolates visual correctness from factual correctness.
  arXiv  Detail & Related papers  (2026-06-02T04:36:32Z)
• PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning [80.94559742826083]
  Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal.<n>Recent work shows that providing a fine-grained, model-intrinsic signal effectively improves language reasoning training by providing step-level guidance without costly external models.<n>While effective for unimodal text, we find that naively applying this global reward to vision-language (V-L) reasoning is a suboptimal strategy.<n>We propose Perception-Decomposed Confidence Reward (PDCR), a framework that solves this by aligning the reward structure with the task's heterogeneous nature.
  arXiv  Detail & Related papers  (2026-05-13T12:55:18Z)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models [53.15468578562038]
  We propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for Multimodal Large Language Models (MLLMs)<n>We first quantify the uncertainty from the model's failure to ground token predictions in the given image.<n>We then encourage more learning pressure on visually deficient tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples.
  arXiv  Detail & Related papers  (2026-05-06T13:08:12Z)
• VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning [5.887179947363831]
  Large Vision Language Models (LVLMs) achieve strong multimodal reasoning but frequently exhibit hallucinations and incorrect responses with high certainty.<n>Existing verbalized confidence calibration methods, largely developed for text-only LLMs, typically optimize a single holistic confidence score using binary answer-level correctness.<n>We propose VL-Calibration, a reinforcement learning framework that explicitly decouples confidence into visual and reasoning confidence.
  arXiv  Detail & Related papers  (2026-04-10T17:47:19Z)
• VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation [22.921677603408188]
  Large Vision-Language Models (LVLMs) frequently hallucinate, limiting their safe deployment in real-world applications.<n>We propose VAUQ, a vision-aware uncertainty quantification framework for LVLM self-evaluation.<n>VAUQ explicitly measures how strongly a model's output depends on visual evidence.
  arXiv  Detail & Related papers  (2026-02-24T16:11:14Z)
• Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering [33.3588824921581]
  We introduce Vision-Language Introspection (VLI), a training-free inference framework that simulates a metacognitive self-correction process.<n>VLI achieves state-of-the-art performance on advanced models, reducing object hallucination rates by 12.67% on MMHal-Bench and improving accuracy by 5.8% on POPE.
  arXiv  Detail & Related papers  (2026-01-08T17:49:13Z)
• Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking [11.763473690046721]
  Reasoning-augmented vision language models generate explicit chains of thought that promise greater capability and transparency.<n>Models may reach correct answers via visually unfaithful intermediate steps, or reason faithfully yet fail on the final prediction.<n>We introduce the visual faithfulness of reasoning chains as a distinct evaluation dimension, focusing on whether the perception steps of a reasoning chain are grounded in the image.
  arXiv  Detail & Related papers  (2025-12-13T07:04:42Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation [26.580361841501514]
  Vision-language models (VLMs) excel in various multimodal tasks but frequently suffer from poor calibration.<n>This miscalibration undermines user trust, especially when models confidently provide incorrect or fabricated information.<n>We propose a novel Confidence through Semantic Perturbation (CSP) framework to improve the calibration of verbalized confidence for object-centric queries.
  arXiv  Detail & Related papers  (2025-04-21T04:01:22Z)
• Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning [67.82016092549284]
  We introduce ReCoVERR, an inference-time algorithm to reduce the over-abstention of a selective vision-language system. ReCoVERR tries to find relevant clues in an image that provide additional evidence for the prediction.
  arXiv  Detail & Related papers  (2024-02-23T21:16:52Z)
• Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting [133.55037976429088]
  We investigate the adversarial robustness of vision transformers equipped with BERT pretraining (e.g., BEiT, MAE) A surprising observation is that MAE has significantly worse adversarial robustness than other BERT pretraining methods. We propose a simple yet effective way to boost the adversarial robustness of MAE.
  arXiv  Detail & Related papers  (2023-08-20T16:27:17Z)
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss [72.62029620566925]
  Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm. Our work analyzes contrastive learning without assuming conditional independence of positive pairs. We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective.
  arXiv  Detail & Related papers  (2021-06-08T07:41:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.