3D-GRES: Generalized 3D Referring Expression Segmentation

• URL: http://arxiv.org/abs/2407.20664v2
• Date: Wed, 31 Jul 2024 11:11:20 GMT
• Title: 3D-GRES: Generalized 3D Referring Expression Segmentation
• Authors: Changli Wu, Yihang Liu, Jiayi Ji, Yiwei Ma, Haowei Wang, Gen Luo, Henghui Ding, Xiaoshuai Sun, Rongrong Ji,
• Abstract summary: 3D Referring Expression (3D-RES) is dedicated to segmenting a specific instance within a 3D space based on a natural language description. Generalized 3D Referring Expression (3D-GRES) extends the capability to segment any number of instances based on natural language instructions.
• Score: 77.10044505645064
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D Referring Expression Segmentation (3D-RES) is dedicated to segmenting a specific instance within a 3D space based on a natural language description. However, current approaches are limited to segmenting a single target, restricting the versatility of the task. To overcome this limitation, we introduce Generalized 3D Referring Expression Segmentation (3D-GRES), which extends the capability to segment any number of instances based on natural language instructions. In addressing this broader task, we propose the Multi-Query Decoupled Interaction Network (MDIN), designed to break down multi-object segmentation tasks into simpler, individual segmentations. MDIN comprises two fundamental components: Text-driven Sparse Queries (TSQ) and Multi-object Decoupling Optimization (MDO). TSQ generates sparse point cloud features distributed over key targets as the initialization for queries. Meanwhile, MDO is tasked with assigning each target in multi-object scenarios to different queries while maintaining their semantic consistency. To adapt to this new task, we build a new dataset, namely Multi3DRes. Our comprehensive evaluations on this dataset demonstrate substantial enhancements over existing models, thus charting a new path for intricate multi-object 3D scene comprehension. The benchmark and code are available at https://github.com/sosppxo/MDIN.

Related papers

• Multimodal 3D Reasoning Segmentation with Complex Scenes [92.92045550692765]
  We bridge the research gaps by proposing a 3D reasoning segmentation task for multiple objects in scenes. The task allows producing 3D segmentation masks and detailed textual explanations as enriched by 3D spatial relations among objects. In addition, we design MORE3D, a simple yet effective method that enables multi-object 3D reasoning segmentation with user questions and textual outputs.
  arXiv  Detail & Related papers  (2024-11-21T08:22:45Z)
• LESS: Label-Efficient and Single-Stage Referring 3D Segmentation [55.06002976797879]
  Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
  arXiv  Detail & Related papers  (2024-10-17T07:47:41Z)
• RefMask3D: Language-Guided Transformer for 3D Referring Segmentation [32.11635464720755]
  RefMask3D aims to explore the comprehensive multi-modal feature interaction and understanding. RefMask3D outperforms previous state-of-the-art method by a large margin of 3.16% mIoU on the challenging ScanRefer dataset.
  arXiv  Detail & Related papers  (2024-07-25T17:58:03Z)
• SegPoint: Segment Any Point Cloud via Large Language Model [62.69797122055389]
  We propose a model, called SegPoint, to produce point-wise segmentation masks across a diverse range of tasks. SegPoint is the first model to address varied segmentation tasks within a single framework.
  arXiv  Detail & Related papers  (2024-07-18T17:58:03Z)
• Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model [108.35777542298224]
  This paper introduces Reason3D, a novel large language model for comprehensive 3D understanding. We propose a hierarchical mask decoder to locate small objects within expansive scenes. Experiments validate that Reason3D achieves remarkable results on large-scale ScanNet and Matterport3D datasets.
  arXiv  Detail & Related papers  (2024-05-27T17:59:41Z)
• PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model [19.333506797686695]
  We introduce a novel segmentation task known as reasoning part segmentation for 3D objects. We output a segmentation mask based on complex and implicit textual queries about specific parts of a 3D object. We propose a model that is capable of segmenting parts of 3D objects based on implicit textual queries and generating natural language explanations.
  arXiv  Detail & Related papers  (2024-04-04T23:38:45Z)
• Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers [65.51132104404051]
  We introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
  arXiv  Detail & Related papers  (2023-12-13T14:27:45Z)
• A One Stop 3D Target Reconstruction and multilevel Segmentation Method [0.0]
  We propose an open-source one stop 3D target reconstruction and multilevel segmentation framework (OSTRA) OSTRA performs segmentation on 2D images, tracks multiple instances with segmentation labels in the image sequence, and then reconstructs labelled 3D objects or multiple parts with Multi-View Stereo (MVS) or RGBD-based 3D reconstruction methods. Our method opens up a new avenue for reconstructing 3D targets embedded with rich multi-scale segmentation information in complex scenes.
  arXiv  Detail & Related papers  (2023-08-14T07:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.