Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?

• URL: http://arxiv.org/abs/2605.30557v1
• Date: Thu, 28 May 2026 20:44:47 GMT
• Title: Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?
• Abstract summary: spatial reasoning is a fundamental capability for vision-language models (VLMs) deployed in real-world environments.<n>We introduce two types of observation challenges: (1) occlusion, which hides target information, and (2) perspective ambiguity, which produces misleading visual cues.<n>For each configuration, we design spatial questions that are answerable under clean observations but require abstention under the introduced challenges.
• Score: 72.2500547961037
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spatial reasoning is a fundamental capability for vision-language models (VLMs) deployed in real-world environments. However, visual observations are inherently limited representations of a 3D world: occlusion can render objects invisible, and perspective can make geometric properties misleading. Despite this, existing spatial reasoning benchmarks typically assume that observations are sufficient and reliable, focusing on whether models produce correct answers rather than whether they recognize when a question cannot be answered and what additional observations would be needed. In this work, we challenge this assumption by constructing a controlled evaluation framework, SpatialUncertain, and introducing two types of observation challenges: (1) occlusion, which hides target information, and (2) perspective ambiguity, which produces misleading visual cues. For each configuration, we design spatial questions that are answerable under clean observations but require abstention under the introduced challenges. We further evaluate whether models can identify which additional viewpoints would resolve perspective ambiguity. Our results across a diverse set of frontier open- and closed-source VLMs reveal two consistent failure modes. First, models are prone to overconfident answering, attempting to solve spatial reasoning tasks even when visual evidence is incomplete or misleading, with average accuracy around 30\% under occlusion and below 10\% under perspective ambiguity. Second, even when additional views are available, some models perform near random chance in identifying which would provide reliable evidence. Together, our findings call for moving beyond answer correctness toward evaluating whether models know when to abstain and how to seek reliable evidence.

Related papers

• MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents [74.83535434786145]
  MNIST-PRO is a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints.<n>We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas.
  arXiv  Detail & Related papers  (2026-08-31T16:05:39Z)
• See2Think: Do Multimodal Models Really Use Intermediate Visual States? [12.082470543980422]
  We introduce See2Think, a unified evaluation framework comprising See2ThinkBench and Visual Action-of-Thought (VAoT)<n>VAoT records textual thoughts, visual actions, rendered states, and subsequent reasoning under four controlled inference settings.<n>We find that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks.
  arXiv  Detail & Related papers  (2026-07-29T11:10:34Z)
• Can Machines Really See Objects in Images? A Study Based on Syntactic Distance and Visual Self-Referential Instances [53.65561607407989]
  We ask whether a model can still classify correctly when such local cues provide no stable basis for distinction.<n>Experiments on ResNets and Vision Transformers reveal a consistent phase-transition phenomenon.<n>Results suggest that general intelligence may require creating new language, not reusing an existing one.
  arXiv  Detail & Related papers  (2026-06-28T14:27:40Z)
• VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning? [79.44137632338062]
  When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence?<n>We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question pairs across cardinality, attribute, spatial, and logic tasks.<n>We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images.
  arXiv  Detail & Related papers  (2026-06-05T22:06:07Z)
• VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence [26.0945130521806]
  Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation.<n>We introduce VISTAQA, a benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering.<n>We introduce GROVE, a metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean.
  arXiv  Detail & Related papers  (2026-05-20T03:44:06Z)
• VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations [47.94531550391802]
  Multi-view visual reasoning is essential for intelligent systems that must understand complex environments from sparse and discrete viewpoints.<n>In real-world scenarios, reasoning across views requires integrating partial observations without explicit guidance.<n>We leverage physically grounded simulation to construct diverse, high-fidelity 3D scenes with precise per-view metadata.
  arXiv  Detail & Related papers  (2026-03-17T13:36:30Z)
• Beyond Accuracy: Evaluating Grounded Visual Evidence in Thinking with Images [34.324634481264034]
  We propose ViEBench, a process-verifiable benchmark designed to evaluate faithful visual reasoning.<n>Comprising 200 multi-scenario high-resolution images with expert-annotated visual evidence, ViEBench categorizes tasks by difficulty into perception and reasoning dimensions.<n>Our experiments yield several interesting observations: (1) VLMs can sometimes produce correct final answers despite grounding on irrelevant regions, and (2) they may successfully locate the correct evidence but still fail to utilize it to reach accurate conclusions.
  arXiv  Detail & Related papers  (2026-01-14T07:25:15Z)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs [7.406217790017003]
  We introduce VisRes Bench, a benchmark to study visual reasoning in naturalistic settings without contextual language supervision.<n>Analyzing model behavior across three levels of complexity, we uncover clear limitations in perceptual and relational visual reasoning capacities.<n>We conclude by discussing how VisRes provides a unified framework for advancing abstract visual reasoning in multimodal research.
  arXiv  Detail & Related papers  (2025-12-24T14:18:38Z)
• VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs [18.349695067647012]
  Visual Language Models excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple tests.<n>We present an evaluation that tests vision-language models' capacity for nonlocal visual reasoning.<n>Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.
  arXiv  Detail & Related papers  (2025-07-04T23:15:52Z)
• Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps [56.76175383189738]
  We introduce ReasonMap, a benchmark designed to assess the fine-grained visual understanding and spatial reasoning abilities of MLLMs.<n>ReasonMap encompasses high-resolution transit maps from 30 cities across 13 countries and includes 1,008 question-answer pairs spanning two question types and three templates.<n> Comprehensive evaluations of 15 popular MLLMs, including both base and reasoning variants, reveal a counterintuitive pattern.
  arXiv  Detail & Related papers  (2025-05-24T12:33:52Z)
• CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting [59.830657530592255]
  Counting Amodally for Patterns Through Unseen REgions (CAPTURe) is a testbed for evaluating vision-language models.<n>We evaluate four strong vision-language models on CAPTURe, finding that models struggle to count on both occluded and unoccluded patterns.
  arXiv  Detail & Related papers  (2025-04-21T23:38:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.