Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models

• URL: http://arxiv.org/abs/2606.02742v1
• Date: Mon, 01 Jun 2026 18:06:08 GMT
• Title: Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models
• Abstract summary: Modern vision-language models (VLMs) remain unreliable on metric distance queries.<n>We introduce textbfViewDiag, a controlled multi-view evaluation protocol built from Hypersim, ScanNet, and KITTI360.<n>We show that stable predictions may reflect prior-driven collapse rather than evidence-sensitive reasoning.
• Score: 29.66841995436342
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spatial reasoning is fundamental to robotics, autonomy, and embodied AI, yet modern vision-language models (VLMs) remain unreliable on metric distance queries. A common assumption is that consistent predictions across viewpoints reflect geometric grounding. We test this assumption and find the opposite: leading VLMs often produce view-invariant and consistent answers even when those answers are incorrect, indicating weak coupling between predictions and viewpoint-specific visual evidence. We introduce \textbf{ViewDiag}, a controlled multi-view evaluation protocol built from Hypersim, ScanNet, and KITTI360, comprising 176 object-pair tracks across 80 scenes with 2--10 views per track. The protocol evaluates models along three axes: metric accuracy, distributional concentration, and a latent feature probe for internal collapse that distinguishes decision collapse from representation collapse. Across diverse models, we observe a consistent pattern of high prediction stability paired with substantial error, clustering in a regime characterized by strong consistency but low accuracy. \noindent These results challenge the common use of cross-view consistency as a proxy for geometric understanding. Instead, we show that stable predictions may reflect prior-driven collapse rather than evidence-sensitive reasoning. ViewDiag provides a controlled benchmark and diagnostic framework for evaluating spatial VLMs beyond accuracy alone. The code and data can be found \href{https://github.com/SDivakarBhat/Consistent_Yet_Wrong.git}{here}

Related papers

• MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents [74.83535434786145]
  MNIST-PRO is a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints.<n>We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas.
  arXiv  Detail & Related papers  (2026-08-31T16:05:39Z)
• Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models [6.612737895321556]
  MultiSymbol Foundation Models are increasingly used as reasoning agents, making reliability, knowing when a model may hallucinate, critical.<n>A common intuition, which we call the Attention-Confidence Assumption, holds that reliability follows from "structural" visual perception.<n>We challenge this through the VLM Reliability Probe (VRP), a systematic cross-family study of reliability signals in contemporary Vision-Language Models (VLMs).
  arXiv  Detail & Related papers  (2026-06-16T00:58:43Z)
• Why Far Looks Up: Probing Spatial Representation in Vision-Language Models [44.01280555484389]
  Vision-language models (VLMs) achieve strong performance on spatial reasoning benchmarks.<n>But it remains unclear whether this reflects structured 3D understanding or reliance on statistical shortcuts in natural images.<n>We introduce a representation-level analysis framework that constructs minimal contrastive pairs to measure how spatial axes are organized.
  arXiv  Detail & Related papers  (2026-05-28T16:18:01Z)
• Are We Overconfident in Models and Results for Semi-Supervised 3D Medical Image Segmentation? [4.120238673372104]
  Semi-supervised learning has become a dominant paradigm for reducing annotation costs.<n>We argue that the current progress is clouded by a twofold overconfidence problem.<n>We propose a tri-space principled segmentation framework founded on a dual-axis reliability assessment engine.
  arXiv  Detail & Related papers  (2026-05-25T08:16:40Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• Learning Uncertainty from Sequential Internal Dispersion in Large Language Models [52.29267172760918]
  Uncertainty estimation is a promising approach to detect hallucinations in large language models.<n>Recent approaches commonly depend on model internal states to estimate uncertainty.<n>We present Sequential Internal Variance Representation (SIVR), a supervised hallucination detection framework.
  arXiv  Detail & Related papers  (2026-04-17T06:31:29Z)
• K$α$LOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks [4.297070083645049]
  We propose K$$LOS, a unified meta-algorithm that generalizes the "Localization First" principle to dataset quality evaluation.<n>By resolving spatial correspondence before assessing agreement, our framework transforms complex-categorical problems into nominal reliability.
  arXiv  Detail & Related papers  (2026-03-28T08:54:05Z)
• SURE: Semi-dense Uncertainty-REfined Feature Matching [28.68008638977835]
  SURE is a Semi- dense Uncertainty-REfined matching framework that jointly predicts correspondences and their confidence.<n>Our approach in- troduces a novel evidential head for trustworthy coordinate regression, along with a lightweight spatial fusion module.<n>Our method consistently outperforms existing state-of-the-art semi-dense matching models in both accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-03-05T06:53:11Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Test-Time Consistency in Vision Language Models [26.475993408532304]
  Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks.<n>Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs.<n>We propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training.
  arXiv  Detail & Related papers  (2025-06-27T17:09:44Z)
• VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives [84.48039784446166]
  We show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason metrics. Our best performing method, Visual Feature Importance Supervision (VisFIS), outperforms strong baselines on benchmark VQA datasets. Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but not faithful.
  arXiv  Detail & Related papers  (2022-06-22T17:02:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.