Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models

• URL: http://arxiv.org/abs/2603.24721v1
• Date: Wed, 25 Mar 2026 18:46:23 GMT
• Title: Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models
• Abstract summary: spatial reasoning focuses on locating target objects based on spatial relations in 3D scenes.<n>Previous approaches have attempted to inject 3D scene representations into the input space of Large Language Models.<n>We propose QuatRoPE, a novel positional embedding method with an input length that is linear to the number of objects.
• Score: 50.14156501544165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spatial reasoning focuses on locating target objects based on spatial relations in 3D scenes, which plays a crucial role in developing intelligent embodied agents. Due to the limited availability of 3D scene-language paired data, it is challenging to train models with strong reasoning ability from scratch. Previous approaches have attempted to inject 3D scene representations into the input space of Large Language Models (LLMs) and leverage the pretrained comprehension and reasoning abilities for spatial reasoning. However, models encoding absolute positions struggle to extract spatial relations from prematurely fused features, while methods explicitly encoding all spatial relations (which is quadratic in the number of objects) as input tokens suffer from poor scalability. To address these limitations, we propose QuatRoPE, a novel positional embedding method with an input length that is linear to the number of objects, and explicitly calculates pairwise spatial relations through the dot product in attention layers. QuatRoPE's holistic vector encoding of 3D coordinates guarantees a high degree of spatial consistency, maintaining fidelity to the scene's geometric integrity. Additionally, we introduce the Isolated Gated RoPE Extension (IGRE), which effectively limits QuatRoPE's influence to object-related tokens, thereby minimizing interference with the LLM's existing positional embeddings and maintaining the LLM's original capabilities. Extensive experiments demonstrate the effectiveness of our approaches. The code and data are available at https://github.com/oceanflowlab/QuatRoPE.

Related papers

• SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video [88.79625979053873]
  We propose a novel framework, SpaceEra, for pre-trained vision-language models (VLMs)<n>We extend the original framework into a comprehensive system, termed SpaceEra++, which spans data construction, model design, training optimization, and prompting inference.<n>In addition, to enhance spatial reasoning, we develop SpaceAlign, which enforces pairwise object constraints by jointly exploiting absolute coordinates and relative spatial relations.
  arXiv  Detail & Related papers  (2026-07-02T06:56:29Z)
• Point Cloud as a Foreign Language for Multi-modal Large Language Model [22.63554774942836]
  SAGE is the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder.<n>Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens.
  arXiv  Detail & Related papers  (2026-03-10T04:22:40Z)
• SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs [21.891285551179365]
  We introduce Spherical Coordinate-based Positional Embedding (SoPE)<n>Our method maps point-cloud token indices into a 3D spherical coordinate space, enabling unified modeling of spatial locations and directional angles.<n>This formulation preserves the inherent geometric structure of point-cloud data, enhances spatial awareness, and yields more consistent and expressive geometric representations for multimodal learning.
  arXiv  Detail & Related papers  (2026-02-26T07:42:15Z)
• Disc3D: Automatic Curation of High-Quality 3D Dialog Data via Discriminative Object Referring [10.067200639268695]
  3D Multi-modal Large Language Models (MLLMs) still lag behind their 2D peers, largely because large-scale, high-quality 3D scene-dialogue datasets remain scarce.<n>We present a fully automated pipeline that converts raw 3D scans into unambiguous, high-quality dialogue data at a fraction of the previous cost.
  arXiv  Detail & Related papers  (2025-11-24T06:51:34Z)
• Video Spatial Reasoning with Object-Centric 3D Rollout [58.12446467377404]
  We propose Object-Centric 3D Rollout (OCR) to enable robust video spatial reasoning.<n>OCR introduces structured perturbations to the 3D geometry of selected objects during training.<n>OCR compels the model to reason holistically across the entire scene.
  arXiv  Detail & Related papers  (2025-11-17T09:53:41Z)
• Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model [51.02616473941499]
  3D object segmentation with Large Language Models (LLMs) has become a prevailing paradigm due to its broad semantics, task flexibility, and strong generalization.<n>However, this paradigm is hindered by representation misalignment: LLMs process high-level semantic tokens, whereas 3D point clouds convey only dense geometric structures.<n>We present the Point Linguist Model (PLM), a general framework that bridges the representation gap between LLMs and dense 3D point clouds.
  arXiv  Detail & Related papers  (2025-09-09T15:01:28Z)
• SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes [105.8644620467576]
  We introduce Stextscurprise3D, a novel dataset designed to evaluate language-guided spatial reasoning segmentation in complex 3D scenes.<n>Stextscurprise3D consists of more than 200k vision language pairs across 900+ detailed indoor scenes from ScanNet++ v2.<n>The dataset contains 89k+ human-annotated spatial queries deliberately crafted without object name.
  arXiv  Detail & Related papers  (2025-07-10T14:01:24Z)
• LLMI3D: MLLM-based 3D Perception from a Single 2D Image [77.13869413871028]
  multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks.<n>In this paper, we propose solutions for weak 3D local spatial object perception, poor text-based geometric numerical output, and inability to handle camera focal variations.<n>We employ parameter-efficient fine-tuning for a pre-trained MLLM and develop LLMI3D, a powerful 3D perception MLLM.
  arXiv  Detail & Related papers  (2024-08-14T10:00:16Z)
• Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3D [71.11034329713058]
  Existing datasets lack large-scale, high-quality 3D ground truth information. Rel3D is the first large-scale, human-annotated dataset for grounding spatial relations in 3D. We propose minimally contrastive data collection -- a novel crowdsourcing method for reducing dataset bias.
  arXiv  Detail & Related papers  (2020-12-03T01:51:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.