Do 3D Large Language Models Really Understand 3D Spatial Relationships?

• URL: http://arxiv.org/abs/2603.23523v1
• Date: Fri, 06 Mar 2026 16:04:34 GMT
• Title: Do 3D Large Language Models Really Understand 3D Spatial Relationships?
• Abstract summary: 3D Large-Language Models claim to understand 3D worlds, especially spatial relationships among objects.<n>We find that fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input.<n>We introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning.
• Score: 80.64317885117704
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding. Project page: https://real-3dqa.github.io/.

Related papers

• ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA [26.947027118604368]
  textbfViewMind3D is a fully training-free and modular framework for 3D spatial reasoning over multi-view observations of a scene.<n>We show that ViewMind3D achieves competitive performance compared to prior training-free and fine-tuned 3D-LLMs.
  arXiv  Detail & Related papers  (2026-07-30T16:14:30Z)
• N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models [45.008146973701855]
  N3D-VLM is a novel unified framework that seamlessly integrates native 3D object perception with 3D-aware visual reasoning.<n>Unlike conventional end-to-end models that directly predict answers from RGB/RGB-D inputs, our approach equips the model with native 3D object perception capabilities.
  arXiv  Detail & Related papers  (2025-12-18T14:03:44Z)
• LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight [105.9472902251177]
  We present a VLM-native recipe that casts 3D detection as a next-token prediction problem.<n>Our model achieves state-of-the-art results, with 49.89 AP_3D, surpassing the previous best by +15.51 absolute improvement.
  arXiv  Detail & Related papers  (2025-11-25T18:59:45Z)
• Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs [72.11701578308804]
  This paper categorizes recent 3D Vision-Language Models into 3D object-centric, 2D image-based, and 3D scene-centric approaches.<n>Despite the architectural similarity of 3D scene-centric VLMs to their 2D counterparts, they have exhibited comparatively lower performance compared with the latest 3D object-centric and 2D image-based approaches.<n>Our investigation suggests that while these models possess cross-modal alignment capabilities, they tend to over-rely on linguistic cues and overfit to frequent answer distributions.
  arXiv  Detail & Related papers  (2025-06-05T17:56:12Z)
• Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D [68.23391872643268]
  LOCATE 3D is a model for localizing objects in 3D scenes from referring expressions like "the small coffee table between the sofa and the lamp"<n>It operates directly on sensor observation streams (posed RGB-D frames), enabling real-world deployment on robots and AR devices.
  arXiv  Detail & Related papers  (2025-04-19T02:51:24Z)
• SplatTalk: 3D VQA with Gaussian Splatting [13.211810095081159]
  Language-guided 3D scene understanding is important for advancing applications in robotics, AR/VR, and human-computer interaction.<n>We introduce SplatTalk, a novel method that uses a generalizable 3D Gaussian Splatting (3DGS) framework to produce 3D tokens suitable for direct input into a pretrained LLM.
  arXiv  Detail & Related papers  (2025-03-08T16:31:48Z)
• Transcrib3D: 3D Referring Expression Resolution through Large Language Models [28.121606686759225]
  We introduce Transcrib3D, an approach that brings together 3D detection methods and the emergent reasoning capabilities of large language models. Transcrib3D achieves state-of-the-art results on 3D reference resolution benchmarks. We show that our method enables a real robot to perform pick-and-place tasks given queries that contain challenging referring expressions.
  arXiv  Detail & Related papers  (2024-04-30T02:48:20Z)
• Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes [56.727745047799246]
  3D scene understanding has gained significant attention due to its wide range of applications. This paper presents Chat-3D, which combines the 3D visual perceptual ability of pre-trained 3D representations and the impressive reasoning and conversation capabilities of advanced LLMs.
  arXiv  Detail & Related papers  (2023-08-17T03:52:15Z)
• 3D-LLM: Injecting the 3D World into Large Language Models [60.43823088804661]
  Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. We propose to inject the 3D world into large language models and introduce a new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks.
  arXiv  Detail & Related papers  (2023-07-24T17:59:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.