LESS: Label-Efficient and Single-Stage Referring 3D Segmentation

• URL: http://arxiv.org/abs/2410.13294v2
• Date: Sat, 26 Oct 2024 05:39:45 GMT
• Title: LESS: Label-Efficient and Single-Stage Referring 3D Segmentation
• Authors: Xuexun Liu, Xiaoxu Xu, Jinlong Li, Qiudan Zhang, Xu Wang, Nicu Sebe, Lin Ma,
• Abstract summary: Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
• Score: 55.06002976797879
• License:
• Abstract: Referring 3D Segmentation is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. Previous works perform a two-stage paradigm, first conducting language-agnostic instance segmentation then matching with given text query. However, the semantic concepts from text query and visual cues are separately interacted during the training, and both instance and semantic labels for each object are required, which is time consuming and human-labor intensive. To mitigate these issues, we propose a novel Referring 3D Segmentation pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. Specifically, we design a Point-Word Cross-Modal Alignment module for aligning the fine-grained features of points and textual embedding. Query Mask Predictor module and Query-Sentence Alignment module are introduced for coarse-grained alignment between masks and query. Furthermore, we propose an area regularization loss, which coarsely reduces irrelevant background predictions on a large scale. Besides, a point-to-point contrastive loss is proposed concentrating on distinguishing points with subtly similar features. Through extensive experiments, we achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels. Code is available at https://github.com/mellody11/LESS.

Related papers

• Open-Vocabulary Octree-Graph for 3D Scene Understanding [54.11828083068082]
  Octree-Graph is a novel scene representation for open-vocabulary 3D scene understanding. An adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape.
  arXiv  Detail & Related papers  (2024-11-25T10:14:10Z)
• DQFormer: Towards Unified LiDAR Panoptic Segmentation with Decoupled Queries [14.435906383301555]
  We propose a novel framework dubbed DQFormer to implement semantic and instance segmentation in a unified workflow. Specifically, we design a decoupled query generator to propose informative queries with semantics by localizing things/stuff positions. We also introduce a query-oriented mask decoder to decode corresponding segmentation masks.
  arXiv  Detail & Related papers  (2024-08-28T14:14:33Z)
• 3D-GRES: Generalized 3D Referring Expression Segmentation [77.10044505645064]
  3D Referring Expression (3D-RES) is dedicated to segmenting a specific instance within a 3D space based on a natural language description. Generalized 3D Referring Expression (3D-GRES) extends the capability to segment any number of instances based on natural language instructions.
  arXiv  Detail & Related papers  (2024-07-30T08:59:05Z)
• SegPoint: Segment Any Point Cloud via Large Language Model [62.69797122055389]
  We propose a model, called SegPoint, to produce point-wise segmentation masks across a diverse range of tasks. SegPoint is the first model to address varied segmentation tasks within a single framework.
  arXiv  Detail & Related papers  (2024-07-18T17:58:03Z)
• SegVG: Transferring Object Bounding Box to Segmentation for Visual Grounding [56.079013202051094]
  We present SegVG, a novel method transfers the box-level annotation as signals to provide an additional pixel-level supervision for Visual Grounding. This approach allows us to iteratively exploit the annotation as signals for both box-level regression and pixel-level segmentation.
  arXiv  Detail & Related papers  (2024-07-03T15:30:45Z)
• EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding [4.447173454116189]
  3D visual grounding aims to find the object within point clouds mentioned by free-form natural language descriptions with rich semantic cues. We present EDA that Explicitly Decouples the textual attributes in a sentence. We further introduce a new visual grounding task, locating objects without object names, which can thoroughly evaluate the model's dense alignment capacity.
  arXiv  Detail & Related papers  (2022-09-29T17:00:22Z)
• Locate then Segment: A Strong Pipeline for Referring Image Segmentation [73.19139431806853]
  Referring image segmentation aims to segment the objects referred by a natural language expression. Previous methods usually focus on designing an implicit and recurrent interaction mechanism to fuse the visual-linguistic features to directly generate the final segmentation mask. We present a "Then-Then-Segment" scheme to tackle these problems. Our framework is simple but surprisingly effective.
  arXiv  Detail & Related papers  (2021-03-30T12:25:27Z)
• SDOD:Real-time Segmenting and Detecting 3D Object by Depth [5.97602869680438]
  This paper proposes a real-time framework that segmenting and detecting 3D objects by depth. We discretize the objects' depth into depth categories and transform the instance segmentation task into a pixel-level classification task. Experiments on the challenging KITTI dataset show that our approach outperforms LklNet about 1.8 times on the speed of segmentation and 3D detection.
  arXiv  Detail & Related papers  (2020-01-26T09:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.