Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

• URL: http://arxiv.org/abs/2605.30161v1
• Date: Thu, 28 May 2026 16:18:01 GMT
• Title: Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
• Abstract summary: Vision-language models (VLMs) achieve strong performance on spatial reasoning benchmarks.<n>But it remains unclear whether this reflects structured 3D understanding or reliance on statistical shortcuts in natural images.<n>We introduce a representation-level analysis framework that constructs minimal contrastive pairs to measure how spatial axes are organized.
• Score: 44.01280555484389
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) achieve strong performance on spatial reasoning benchmarks, yet it remains unclear whether this reflects structured 3D understanding or reliance on statistical shortcuts in natural images. We introduce a representation-level analysis framework that constructs minimal contrastive pairs to measure how spatial axes are organized and disentangled within VLM embeddings. Our analysis across multiple model families reveals a consistent vertical-distance entanglement: models conflate vertical image position with distance, mirroring the perspective bias of natural photographs. This bias produces a significant accuracy gap between perspective-consistent and counter-heuristic examples, and intensifies under data scaling even as overall benchmark accuracy improves. We further show that models with similar benchmark scores can exhibit different internal representations, and that these differences predict accuracy and robustness across diverse spatial reasoning benchmarks. To isolate this bias from evaluation-set skew, we introduce SpatialTunnel, a synthetic benchmark designed to expose spatial shortcut biases by removing common correlations present in natural images. Experiments confirm that the entanglement is model-intrinsic, and that models with well-separated spatial axes exhibit greater robustness, suggesting that well-structured spatial representations lead to more reliable spatial reasoning across diverse benchmarks. Code and benchmark are available on the project page: https://cheolhong0916.github.io/whyfarlooksup.github.io/.

Related papers

• Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models [29.66841995436342]
  Modern vision-language models (VLMs) remain unreliable on metric distance queries.<n>We introduce textbfViewDiag, a controlled multi-view evaluation protocol built from Hypersim, ScanNet, and KITTI360.<n>We show that stable predictions may reflect prior-driven collapse rather than evidence-sensitive reasoning.
  arXiv  Detail & Related papers  (2026-06-01T18:06:08Z)
• SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generation [62.55421542903781]
  We present textbfSpatialReward, a verifiable reward model explicitly designed to evaluate spatial layouts in generated images.<n>Experiments on Stable Diffusion and FLUX show that incorporating SpatialReward into RL training consistently improves spatial consistency and overall generation quality.
  arXiv  Detail & Related papers  (2026-03-23T17:26:35Z)
• Geo-ID: Test-Time Geometric Consensus for Cross-View Consistent Intrinsics [37.614964138575935]
  Intrinsic image decomposition aims to estimate physically based rendering parameters such as albedo, roughness, and metallicity from images.<n>Video-based models can improve cross-frame consistency but require dense, ordered sequences and substantial compute.<n>We propose Geo-ID, a novel test-time framework that repurposes pretrained single-view predictors intrinsic to produce cross-view consistent decompositions.
  arXiv  Detail & Related papers  (2026-03-14T09:36:27Z)
• m2sv: A Scalable Benchmark for Map-to-Street-View Spatial Reasoning [1.2162625261919089]
  We introduce m2sv, a scalable benchmark for map-to-street-view spatial reasoning.<n>We release m2sv-20k, a geographically diverse benchmark with controlled ambiguity, along with m2sv-sft-11k, a curated set of structured reasoning traces for supervised fine-tuning.<n>Despite strong performance on existing multimodal benchmarks, the best evaluated VLM achieves only 65.2% accuracy on m2sv, far below the human baseline of 95%.
  arXiv  Detail & Related papers  (2026-01-27T02:01:56Z)
• Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models [14.442394137843923]
  We present a detailed analysis that first delineates the core elements of spatial reasoning.<n>We then assesses the performance of these models in both synthetic and real-world images.
  arXiv  Detail & Related papers  (2025-03-25T14:34:06Z)
• Linking Robustness and Generalization: A k* Distribution Analysis of Concept Clustering in Latent Space for Vision Models [56.89974470863207]
  This article uses the k* Distribution, a local neighborhood analysis method, to examine the learned latent space at the level of individual concepts. We introduce skewness-based true and approximate metrics for interpreting individual concepts to assess the overall quality of vision models' latent space.
  arXiv  Detail & Related papers  (2024-08-17T01:43:51Z)
• REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models [67.55362046790512]
  Vision-language models lack the ability to correctly reason over spatial relationships. We develop the REVISION framework which improves spatial fidelity in vision-language models. Our results and findings indicate that utilizing rendering-based frameworks is an effective approach for developing spatially-aware models.
  arXiv  Detail & Related papers  (2024-08-05T04:51:46Z)
• Intriguing Equivalence Structures of the Embedding Space of Vision Transformers [1.7418480517632609]
  Pre-trained large foundation models play a central role in the recent surge of artificial intelligence. Due to their inherent complexity, these models are not well understood. We show via analyses and systematic experiments that the representation space consists of large piecewise linear subspaces.
  arXiv  Detail & Related papers  (2024-01-28T04:59:51Z)
• Towards Robust and Expressive Whole-body Human Pose and Shape Estimation [51.457517178632756]
  Whole-body pose and shape estimation aims to jointly predict different behaviors of the entire human body from a monocular image. Existing methods often exhibit degraded performance under the complexity of in-the-wild scenarios. We propose a novel framework to enhance the robustness of whole-body pose and shape estimation.
  arXiv  Detail & Related papers  (2023-12-14T08:17:42Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.