Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels

• URL: http://arxiv.org/abs/2312.17232v1
• Date: Thu, 28 Dec 2023 18:57:11 GMT
• Title: Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels
• Authors: Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, Francis Engelmann
• Abstract summary: Current 3D scene segmentation methods are heavily dependent on manually annotated 3D training datasets. We propose Segment3D, a method for class-agnostic 3D scene segmentation that produces high-quality 3D segmentation masks.
• Score: 141.23836433191624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current 3D scene segmentation methods are heavily dependent on manually annotated 3D training datasets. Such manual annotations are labor-intensive, and often lack fine-grained details. Importantly, models trained on this data typically struggle to recognize object classes beyond the annotated classes, i.e., they do not generalize well to unseen domains and require additional domain-specific annotations. In contrast, 2D foundation models demonstrate strong generalization and impressive zero-shot abilities, inspiring us to incorporate these characteristics from 2D models into 3D models. Therefore, we explore the use of image segmentation foundation models to automatically generate training labels for 3D segmentation. We propose Segment3D, a method for class-agnostic 3D scene segmentation that produces high-quality 3D segmentation masks. It improves over existing 3D segmentation models (especially on fine-grained masks), and enables easily adding new training data to further boost the segmentation performance -- all without the need for manual training labels.

Related papers

• Bayesian Self-Training for Semi-Supervised 3D Segmentation [59.544558398992386]
  3D segmentation is a core problem in computer vision. densely labeling 3D point clouds to employ fully-supervised training remains too labor intensive and expensive. Semi-supervised training provides a more practical alternative, where only a small set of labeled data is given, accompanied by a larger unlabeled set.
  arXiv  Detail & Related papers  (2024-09-12T14:54:31Z)
• Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels [69.55622471172941]
  Large-scale vision 2D vision language models, such as CLIP can be aligned with a 3D encoder to learn generalizable (open-vocabulary) 3D vision models. We propose an optimization framework: Cross-MoST: Cross-Modal Self-Training, to improve the label-free classification performance of a zero-shot 3D vision model.
  arXiv  Detail & Related papers  (2024-04-15T21:30:50Z)
• SAM-guided Graph Cut for 3D Instance Segmentation [60.75119991853605]
  This paper addresses the challenge of 3D instance segmentation by simultaneously leveraging 3D geometric and multi-view image information. We introduce a novel 3D-to-2D query framework to effectively exploit 2D segmentation models for 3D instance segmentation. Our method achieves robust segmentation performance and can generalize across different types of scenes.
  arXiv  Detail & Related papers  (2023-12-13T18:59:58Z)
• DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields [68.94868475824575]
  This paper introduces a novel approach capable of generating infinite, high-quality 3D-consistent 2D annotations alongside 3D point cloud segmentations. We leverage the strong semantic prior within a 3D generative model to train a semantic decoder. Once trained, the decoder efficiently generalizes across the latent space, enabling the generation of infinite data.
  arXiv  Detail & Related papers  (2023-11-18T21:58:28Z)
• UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenes [35.38074724231105]
  UnScene3D is a fully unsupervised 3D learning approach for class-agnostic 3D instance segmentation of indoor scans. We operate on a basis of geometric oversegmentation, enabling efficient representation and learning on high-resolution 3D data. Our approach improves over state-of-the-art unsupervised 3D instance segmentation methods by more than 300% Average Precision score.
  arXiv  Detail & Related papers  (2023-03-25T19:15:16Z)
• MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation [91.6658845016214]
  We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. We render a 3D shape from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent.
  arXiv  Detail & Related papers  (2022-08-18T00:48:15Z)
• Semi-supervised 3D shape segmentation with multilevel consistency and part substitution [21.075426681857024]
  We propose an effective semi-supervised method for learning 3D segmentations from a few labeled 3D shapes and a large amount of unlabeled 3D data. For the unlabeled data, we present a novel multilevel consistency loss to enforce consistency of network predictions between perturbed copies of a 3D shape. For the labeled data, we develop a simple yet effective part substitution scheme to augment the labeled 3D shapes with more structural variations to enhance training.
  arXiv  Detail & Related papers  (2022-04-19T11:48:24Z)
• Interactive Object Segmentation in 3D Point Clouds [27.88495480980352]
  We present an interactive 3D object segmentation method in which the user interacts directly with the 3D point cloud. Our model does not require training data from the target domain. It performs well on several other datasets with different data characteristics as well as different object classes.
  arXiv  Detail & Related papers  (2022-04-14T18:31:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.