Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes

• URL: http://arxiv.org/abs/2206.01203v3
• Date: Tue, 31 Oct 2023 16:50:00 GMT
• Title: Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes
• Authors: Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll
• Abstract summary: We look at weakly-supervised 3D semantic instance segmentation. Key idea is to leverage 3D bounding box labels which are easier and faster to annotate. We show that it is possible to train dense segmentation models using only bounding box labels.
• Score: 38.60444957213202
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current 3D segmentation methods heavily rely on large-scale point-cloud datasets, which are notoriously laborious to annotate. Few attempts have been made to circumvent the need for dense per-point annotations. In this work, we look at weakly-supervised 3D semantic instance segmentation. The key idea is to leverage 3D bounding box labels which are easier and faster to annotate. Indeed, we show that it is possible to train dense segmentation models using only bounding box labels. At the core of our method, \name{}, lies a deep model, inspired by classical Hough voting, that directly votes for bounding box parameters, and a clustering method specifically tailored to bounding box votes. This goes beyond commonly used center votes, which would not fully exploit the bounding box annotations. On ScanNet test, our weakly supervised model attains leading performance among other weakly supervised approaches (+18 mAP@50). Remarkably, it also achieves 97% of the mAP@50 score of current fully supervised models. To further illustrate the practicality of our work, we train Box2Mask on the recently released ARKitScenes dataset which is annotated with 3D bounding boxes only, and show, for the first time, compelling 3D instance segmentation masks.

Related papers

• Bayesian Self-Training for Semi-Supervised 3D Segmentation [59.544558398992386]
  3D segmentation is a core problem in computer vision. densely labeling 3D point clouds to employ fully-supervised training remains too labor intensive and expensive. Semi-supervised training provides a more practical alternative, where only a small set of labeled data is given, accompanied by a larger unlabeled set.
  arXiv  Detail & Related papers  (2024-09-12T14:54:31Z)
• When 3D Bounding-Box Meets SAM: Point Cloud Instance Segmentation with Weak-and-Noisy Supervision [20.625754683390536]
  We propose a complementary image prompt-induced weakly-supervised point cloud instance segmentation (CIP-WPIS) method. We leverage pretrained knowledge embedded in the 2D foundation model SAM and 3D geometric prior to achieve accurate point-wise instance labels. Our method is robust against noisy 3D bounding-box annotations and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-09-02T05:17:03Z)
• GaPro: Box-Supervised 3D Point Cloud Instance Segmentation Using Gaussian Processes as Pseudo Labelers [14.88505076974645]
  GaPro is a new instance segmentation for 3D point clouds using axis-aligned 3D bounding box supervision. Our two-step approach involves generating pseudo labels from box annotations and training a 3DIS network with the resulting labels. Our experiments show that GaPro outperforms previous weakly supervised 3D instance segmentation methods.
  arXiv  Detail & Related papers  (2023-07-25T04:43:22Z)
• You Only Need One Thing One Click: Self-Training for Weakly Supervised 3D Scene Understanding [107.06117227661204]
  We propose One Thing One Click'', meaning that the annotator only needs to label one point per object. We iteratively conduct the training and label propagation, facilitated by a graph propagation module. Our model can be compatible to 3D instance segmentation equipped with a point-clustering strategy.
  arXiv  Detail & Related papers  (2023-03-26T13:57:00Z)
• BoxSnake: Polygonal Instance Segmentation with Box Supervision [34.487089567665556]
  We propose a new end-to-end training technique, termed BoxSnake, to achieve effective polygonal instance segmentation using only box annotations for the first time. Compared with the mask-based weakly-supervised methods, BoxSnake further reduces the performance gap between the predicted segmentation and the bounding box, and shows significant superiority on the Cityscapes dataset.
  arXiv  Detail & Related papers  (2023-03-21T06:54:18Z)
• Weakly Supervised Monocular 3D Object Detection using Multi-View Projection and Direction Consistency [78.76508318592552]
  Monocular 3D object detection has become a mainstream approach in automatic driving for its easy application. Most current methods still rely on 3D point cloud data for labeling the ground truths used in the training phase. We propose a new weakly supervised monocular 3D objection detection method, which can train the model with only 2D labels marked on images.
  arXiv  Detail & Related papers  (2023-03-15T15:14:00Z)
• Collaborative Propagation on Multiple Instance Graphs for 3D Instance Segmentation with Single-point Supervision [63.429704654271475]
  We propose a novel weakly supervised method RWSeg that only requires labeling one object with one point. With these sparse weak labels, we introduce a unified framework with two branches to propagate semantic and instance information. Specifically, we propose a Cross-graph Competing Random Walks (CRW) algorithm that encourages competition among different instance graphs.
  arXiv  Detail & Related papers  (2022-08-10T02:14:39Z)
• Box2Seg: Learning Semantics of 3D Point Clouds with Box-Level Supervision [65.19589997822155]
  We introduce a neural architecture, termed Box2Seg, to learn point-level semantics of 3D point clouds with bounding box-level supervision. We show that the proposed network can be trained with cheap, or even off-the-shelf bounding box-level annotations and subcloud-level tags.
  arXiv  Detail & Related papers  (2022-01-09T09:07:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.