Do Vision--Language Models Understand 3D Scenes or Just Catalogue Objects?

• URL: http://arxiv.org/abs/2605.20448v1
• Date: Tue, 19 May 2026 20:01:19 GMT
• Title: Do Vision--Language Models Understand 3D Scenes or Just Catalogue Objects?
• Abstract summary: 6 frontier and open-weight VLMs, scored by trained annotators on 18,204 responses with no LLM-as-judge, reveal a sharp dissociation.<n>Models that plan rearrangements over visible layouts at 53--97% accuracy and rarely violate collision constraints fall to 6--45% on occlusion and below 7% on reflections.<n>White-box analysis on Qwen3-VL-8B-Thinking localises the failure to the visual-token merger.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision--language models reliably name objects in a scene, but do they represent the 3D layout those objects inhabit? We introduce a 3,034-sample human-curated benchmark targeting three components of spatial understanding: depth-ordered occlusion (probed via three independent counterfactual operationalisations), optical-geometry inference over visible reflections, and volumetric rearrangement planning. Six frontier and open-weight VLMs, scored by trained annotators on 18,204 responses with no LLM-as-judge, reveal a sharp dissociation: models that plan rearrangements over visible layouts at 53--97% accuracy and rarely violate collision constraints fall to 6--45% on occlusion and below 7% on reflections. An embodied-reasoning model reproduces the same profile. White-box analysis on Qwen3-VL-8B-Thinking localises the failure to the visual-token merger: spatial information recoverable throughout the vision encoder becomes inaccessible after token compression and only stabilises again when clean post-merger activations are patched into the language decoder.

Related papers

• Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Models [9.514456645829966]
  We show that visual encoders exhibit attention artifacts previously documented in generic Vision Transformers.<n>We introduce AtVLA, a framework that inserts learnable register tokens into the visual encoder.
  arXiv  Detail & Related papers  (2026-08-03T13:18:36Z)
• VLRC: Vision-Language Reprojection Consistency as a scalable signal for better feed-forward 3D pretraining [8.146936489399293]
  Vision-Language Reprojection Consistency (VLRC) exploits frozen vision-language representations as semantic multi-view supervision.<n>VLRC integrates seamlessly with both self-supervised monocular reconstruction and supervised-pretrained feed-forward 3D models.<n>Experiments on indoor and outdoor benchmarks demonstrate consistent gains in 3D reconstruction accuracy and zero-shot open-vocabulary 3D semantic segmentation.
  arXiv  Detail & Related papers  (2026-07-02T18:49:34Z)
• 3D-DLP: Self-Supervised 3D Object-Centric Scene Representation Learning [37.932415885988924]
  We introduce 3D-DLP, a self-supervised object-centric representation learning model that decomposes scene-level RGB-D or voxel observations into a set of 3D latent particles.<n>The model learns interpretable per-particle segmentation maps through an end-to-end self-supervised reconstruction objective.<n>We show that leveraging these compact 3D latent particles for downstream robotic manipulation improves performance over baselines.
  arXiv  Detail & Related papers  (2026-06-17T18:00:08Z)
• Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agents [52.53348718474685]
  Vision-Language Models (VLMs) exhibit strong visual reasoning capabilities, yet they still struggle with 3D understanding.<n>We introduce three inference-time techniques that are essential to this closed-loop process.<n>Our approach surpasses prior methods at predicting the text-guided goal 6D pose of the target object.
  arXiv  Detail & Related papers  (2026-04-10T18:06:02Z)
• Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models [50.14156501544165]
  spatial reasoning focuses on locating target objects based on spatial relations in 3D scenes.<n>Previous approaches have attempted to inject 3D scene representations into the input space of Large Language Models.<n>We propose QuatRoPE, a novel positional embedding method with an input length that is linear to the number of objects.
  arXiv  Detail & Related papers  (2026-03-25T18:46:23Z)
• Video Spatial Reasoning with Object-Centric 3D Rollout [58.12446467377404]
  We propose Object-Centric 3D Rollout (OCR) to enable robust video spatial reasoning.<n>OCR introduces structured perturbations to the 3D geometry of selected objects during training.<n>OCR compels the model to reason holistically across the entire scene.
  arXiv  Detail & Related papers  (2025-11-17T09:53:41Z)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models [100.13033631690114]
  Vision-language models (VLMs) struggle with 3D-related tasks such as spatial cognition and physical understanding.<n>We introduce SandboxVLM, a framework that leverages abstract bounding boxes to encode geometric structure and physical kinematics for VLM.<n>Our approach consistently improves spatial intelligence, achieving an 8.3% gain on SAT Real compared with baseline methods.
  arXiv  Detail & Related papers  (2025-11-14T04:16:09Z)
• Evaluation of Vision-LLMs in Surveillance Video [8.750453732584491]
  This paper investigates the spatial reasoning of vision-language models (VLMs)<n>It addresses the embodied perception challenge of interpreting dynamic 3D scenes from sparse 2D video.<n>We evaluate four open models on UCF-Crime and RWF-2000 under prompting and privacy-preserving conditions.
  arXiv  Detail & Related papers  (2025-10-27T10:27:02Z)
• Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model [51.02616473941499]
  3D object segmentation with Large Language Models (LLMs) has become a prevailing paradigm due to its broad semantics, task flexibility, and strong generalization.<n>However, this paradigm is hindered by representation misalignment: LLMs process high-level semantic tokens, whereas 3D point clouds convey only dense geometric structures.<n>We present the Point Linguist Model (PLM), a general framework that bridges the representation gap between LLMs and dense 3D point clouds.
  arXiv  Detail & Related papers  (2025-09-09T15:01:28Z)
• Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames [17.975173937253494]
  An embodied AI assistant operating on egocentric video must integrate spatial cues across time.<n>Disjoint-3DQA is a generative QA benchmark that evaluates this ability of VLMs.
  arXiv  Detail & Related papers  (2025-05-30T06:32:26Z)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction [86.82819259860186]
  We introduce VLM-3R, a unified framework for Vision-Language Models (VLMs) that incorporates 3D Reconstructive instruction tuning.<n>VLM-3R processes monocular video frames by employing a geometry encoder to derive implicit 3D tokens that represent spatial understanding.
  arXiv  Detail & Related papers  (2025-05-26T17:56:30Z)
• Unified Scene Representation and Reconstruction for 3D Large Language Models [40.693839066536505]
  Existing approaches extract point clouds either from ground truth (GT) geometry or 3D scenes reconstructed by auxiliary models. We introduce Uni3DR2 extracts 3D geometric and semantic aware representation features via the frozen 2D foundation models. Our learned 3D representations not only contribute to the reconstruction process but also provide valuable knowledge for LLMs.
  arXiv  Detail & Related papers  (2024-04-19T17:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.