PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding

• URL: http://arxiv.org/abs/2606.06485v1
• Date: Thu, 04 Jun 2026 17:59:04 GMT
• Title: PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding
• Abstract summary: We present PAR3D, a unified part-aware 3D-MLLM framework that enables models to understand, reason about, and ground both objects and their parts in 3D scenes.<n>Our method substantially improves part-level question answering and referring segmentation, while also achieving strong performance across object-level vision-language tasks.
• Score: 41.210392758568126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in 3D multimodal large language models (3D-MLLMs) have enabled unified solutions for 3D scene understanding tasks, including visual question answering, captioning, and referring segmentation. However, existing 3D-MLLMs remain largely object-centric, limiting their ability to model fine-grained part structures that are essential for embodied interaction with 3D environments. In this work, we present PAR3D, a unified part-aware 3D-MLLM framework that enables models to understand, reason about, and ground both objects and their parts in 3D scenes. To enable training and evaluation of part-aware 3D scene understanding, we introduce ScenePart, a synthetic 3D scene dataset with part-level annotations and language instructions. We further develop Part-Aware 3D Representation Learning to enrich 3D visual representations with fine-grained part-level semantics, and propose Hierarchical Segmentation Query Generation to ground part targets via hierarchical object-part queries. Extensive experiments show that our method substantially improves part-level question answering and referring segmentation, while also achieving strong performance across object-level vision-language tasks.

Related papers

• Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM [71.01178857331969]
  Chat-Scene++ is an MLLM framework that represents 3D scenes as context-rich object sequences.<n>By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction.<n>Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks.
  arXiv  Detail & Related papers  (2026-03-29T04:16:04Z)
• Segment Any 3D-Part in a Scene from a Sentence [50.46950922754459]
  This paper aims to achieve the segmentation of any 3D part in a scene based on natural language descriptions.<n>We introduce the 3D-PU dataset, the first large-scale 3D dataset with dense part annotations.<n>On the methodological side, we propose OpenPart3D, a 3D-input-only framework to tackle the challenges of part-level segmentation.
  arXiv  Detail & Related papers  (2025-06-24T05:51:22Z)
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation [91.94869042117621]
  Reasoning segmentation aims to segment target objects in complex scenes based on human intent and spatial reasoning.<n>Recent multimodal large language models (MLLMs) have demonstrated impressive 2D image reasoning segmentation.<n>We introduce MLLM-For3D, a framework that transfers knowledge from 2D MLLMs to 3D scene understanding.
  arXiv  Detail & Related papers  (2025-03-23T16:40:20Z)
• Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description [56.69740649781989]
  3D scene understanding is a long-standing challenge in computer vision and a key component in enabling mixed reality, wearable computing, and embodied AI.<n>We introduce Articulate3D, an expertly curated 3D dataset featuring high-quality manual annotations on 280 indoor scenes.<n>We also present USDNet, a novel unified framework capable of simultaneously predicting part segmentation along with a full specification of motion attributes for articulated objects.
  arXiv  Detail & Related papers  (2024-12-02T11:33:55Z)
• Kestrel: 3D Multimodal LLM for Part-Aware Grounded Description [33.55332803244455]
  Part-Aware Point Grounded Description (PaPGD) is a challenging task aimed at advancing 3D multimodal learning for fine-grained, part-aware segmentation grounding.<n>We present the 3DCoMPaT Grounded Instructions (3DCoMPaT-GrIn) dataset, a comprehensive resource that pairs rich point cloud descriptions with corresponding part-level segmentation masks.<n>We propose Kestrel, a part-aware 3D multimodal large language model that integrates an advanced language model for nuanced language comprehension with multi-level point feature propagation and query refinement mechanism.
  arXiv  Detail & Related papers  (2024-05-29T09:43:48Z)
• 3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding [12.823274886850697]
  We introduce a novel and efficient prompt tuning paradigm, 3DMIT. This paradigm eliminates the alignment stage between 3D scenes and language and extends the instruction prompt with the 3D modality information. We evaluate the effectiveness of our method across diverse tasks in the 3D scene domain.
  arXiv  Detail & Related papers  (2024-01-06T12:20:18Z)
• Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers [65.51132104404051]
  We introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
  arXiv  Detail & Related papers  (2023-12-13T14:27:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.