SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

• URL: http://arxiv.org/abs/2609.16233v1
• Date: Mon, 14 Sep 2026 18:59:13 GMT
• Title: SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
• Abstract summary: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning.<n>We introduce SceneBench, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting.<n>SceneBench provides a realistic testbed for developing and evaluating models capable of fine-grained spatial reasoning.
• Score: 12.672496253081258
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benchmarks. First, 3D datasets often rely on point clouds that capture geometry but discard rich visual features like texture, text, and materials. Second, annotations treat objects in isolation while ignoring real-world hierarchical organization (scenes, rooms, functional areas, object groups). Third, evaluation tasks focus narrowly on basic recognition rather than multi-step spatial reasoning. In this context, we introduce SceneBench, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting and densely annotated with hierarchical semantics spanning scenes, rooms, functional areas, object groups, and individual objects. These annotations are produced through a human-in-the-loop pipeline combining vision-language models with roughly 1,500 human-hours of iterative refinement and verification, producing over 183K annotated nodes with textual descriptions and 3D bounding boxes. Building on this representation, we define three evaluation tasks: Existence-Based Questions probing object attributes, Spatial Intelligence Questions covering counting, size comparison, distance, and directional relations, and Grounded Question-Reasoning-Answer (QRA) triplets requiring multi-step reasoning across semantic levels. Experiments with state-of-the-art vision-language models show that while models perform well on basic recognition tasks (e.g., up to 85% accuracy for detection), performance drops substantially on hierarchical and compositional reasoning (e.g., down to 60% for counting), revealing limitations not captured by existing benchmarks. SceneBench provides a realistic testbed for developing and evaluating models capable of fine-grained spatial reasoning in photorealistic 3D environments.

Related papers

• Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM [71.01178857331969]
  Chat-Scene++ is an MLLM framework that represents 3D scenes as context-rich object sequences.<n>By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction.<n>Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks.
  arXiv  Detail & Related papers  (2026-03-29T04:16:04Z)
• RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roaming [79.81527946524098]
  RoamScene3D is a novel framework that bridges the gap between semantic guidance and spatial generation.<n>We employ a vision-language model (VLM) to construct a scene graph that encodes object relations.<n>To mitigate the limitations of static 2D priors, we introduce a Motion-Injected Inpainting model that is fine-tuned on a synthetic panoramic dataset.
  arXiv  Detail & Related papers  (2026-01-27T10:10:55Z)
• TUN3D: Towards Real-World Scene Understanding from Unposed Images [11.23080017635425]
  TUN3D is a new method that tackles joint layout estimation and 3D object detection in real scans.<n>It does not require ground-truth camera poses or depth supervision.<n>It achieves state-of-the-art performance across three challenging scene understanding benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T20:24:07Z)
• Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions [28.185661905201222]
  Descrip3D is a novel framework that explicitly encodes the relationships between objects using natural language.<n>It allows for unified reasoning across various tasks such as grounding, captioning, and question answering.
  arXiv  Detail & Related papers  (2025-07-19T09:19:16Z)
• SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes [105.8644620467576]
  We introduce Stextscurprise3D, a novel dataset designed to evaluate language-guided spatial reasoning segmentation in complex 3D scenes.<n>Stextscurprise3D consists of more than 200k vision language pairs across 900+ detailed indoor scenes from ScanNet++ v2.<n>The dataset contains 89k+ human-annotated spatial queries deliberately crafted without object name.
  arXiv  Detail & Related papers  (2025-07-10T14:01:24Z)
• From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes [30.015378490907988]
  Anywhere3D-Bench is a holistic 3D visual grounding benchmark consisting of 2,886 referring expression-3D bounding box pairs.<n>We assess a range of state-of-the-art 3D visual grounding methods alongside large language models.
  arXiv  Detail & Related papers  (2025-06-05T11:28:02Z)
• Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models [14.442394137843923]
  We present a detailed analysis that first delineates the core elements of spatial reasoning.<n>We then assesses the performance of these models in both synthetic and real-world images.
  arXiv  Detail & Related papers  (2025-03-25T14:34:06Z)
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [55.022519020409405]
  This paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan.<n>The resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T17:59:30Z)
• Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers [65.51132104404051]
  We introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
  arXiv  Detail & Related papers  (2023-12-13T14:27:45Z)
• 3D Concept Grounding on Neural Fields [99.33215488324238]
  Existing visual reasoning approaches typically utilize supervised methods to extract 2D segmentation masks on which concepts are grounded. Humans are capable of grounding concepts on the underlying 3D representation of images. We propose to leverage the continuous, differentiable nature of neural fields to segment and learn concepts.
  arXiv  Detail & Related papers  (2022-07-13T17:59:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.