Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

• URL: http://arxiv.org/abs/2603.27507v1
• Date: Sun, 29 Mar 2026 04:16:04 GMT
• Title: Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM
• Abstract summary: Chat-Scene++ is an MLLM framework that represents 3D scenes as context-rich object sequences.<n>By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction.<n>Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks.
• Score: 71.01178857331969
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in multi-modal large language models (MLLMs) have shown strong potential for 3D scene understanding. However, existing methods struggle with fine-grained object grounding and contextual reasoning, limiting their ability to interpret and interact with complex 3D environments. In this paper, we present Chat-Scene++, an MLLM framework that represents 3D scenes as context-rich object sequences. By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction. It decomposes a 3D scene into object representations paired with identifier tokens, allowing LLMs to follow instructions across diverse 3D vision-language tasks. To capture inter-object relationships and global semantics, Chat-Scene++ extracts context-rich object features using large-scale pre-trained 3D scene-level and 2D image-level encoders, unlike the isolated per-object features in Chat-Scene. Its flexible object-centric design also supports grounded chain-of-thought (G-CoT) reasoning, enabling the model to distinguish objects at both category and spatial levels during multi-step inference. Without the need for additional task-specific heads or fine-tuning, Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks: ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D. These results highlight its effectiveness in scene comprehension, object grounding, and spatial reasoning. Additionally, without reconstructing 3D worlds through computationally expensive processes, we demonstrate its applicability to real-world scenarios using only 2D inputs.

Related papers

• Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understanding [38.97818584066075]
  Text-Scene is a framework that automatically parses 3D scenes into textual descriptions for scene understanding.<n>By leveraging both geometric analysis and MLLMs, Text-Scene produces descriptions that are accurate, detailed, and human-interpretable.
  arXiv  Detail & Related papers  (2025-09-20T15:10:45Z)
• Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions [28.185661905201222]
  Descrip3D is a novel framework that explicitly encodes the relationships between objects using natural language.<n>It allows for unified reasoning across various tasks such as grounding, captioning, and question answering.
  arXiv  Detail & Related papers  (2025-07-19T09:19:16Z)
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation [91.94869042117621]
  Reasoning segmentation aims to segment target objects in complex scenes based on human intent and spatial reasoning.<n>Recent multimodal large language models (MLLMs) have demonstrated impressive 2D image reasoning segmentation.<n>We introduce MLLM-For3D, a framework that transfers knowledge from 2D MLLMs to 3D scene understanding.
  arXiv  Detail & Related papers  (2025-03-23T16:40:20Z)
• 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding [0.5755004576310334]
  A 3D scene graph represents a compact scene model by capturing both the objects present and the semantic relationships between them.<n>We propose 3DGraphLLM, a method for constructing a learnable representation of a 3D scene graph that explicitly incorporates semantic relationships.
  arXiv  Detail & Related papers  (2024-12-24T14:21:58Z)
• ChatSplat: 3D Conversational Gaussian Splatting [51.40403199909113]
  ChatSplat is a system that constructs a 3D language field, enabling rich chat-based interaction within 3D space.<n>For view-level interaction, we designed an encoder that encodes the rendered feature map of each view into tokens, which are then processed by a large language model.<n>At the scene level, ChatSplat combines multi-view tokens, enabling interactions that consider the entire scene.
  arXiv  Detail & Related papers  (2024-12-01T08:59:30Z)
• Multimodal 3D Reasoning Segmentation with Complex Scenes [92.92045550692765]
  We propose a 3D reasoning segmentation task for reasoning segmentation with multiple objects in scenes.<n>The task allows producing 3D segmentation masks and detailed textual explanations as enriched by 3D spatial relations among objects.<n>In addition, we design MORE3D, a novel 3D reasoning network that works with queries of multiple objects.
  arXiv  Detail & Related papers  (2024-11-21T08:22:45Z)
• Grounded 3D-LLM with Referent Tokens [58.890058568493096]
  We propose Grounded 3D-LLM to consolidate various 3D vision tasks within a unified generative framework. The model uses scene referent tokens as special noun phrases to reference 3D scenes. Per-task instruction-following templates are employed to ensure natural and diversity in translating 3D vision tasks into language formats.
  arXiv  Detail & Related papers  (2024-05-16T18:03:41Z)
• Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers [65.51132104404051]
  We introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
  arXiv  Detail & Related papers  (2023-12-13T14:27:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.