Data-Efficient Semantic Segmentation of 3D Point Clouds via Open-Vocabulary Image Segmentation-based Pseudo-Labeling

• URL: http://arxiv.org/abs/2604.11007v1
• Date: Mon, 13 Apr 2026 05:16:39 GMT
• Title: Data-Efficient Semantic Segmentation of 3D Point Clouds via Open-Vocabulary Image Segmentation-based Pseudo-Labeling
• Abstract summary: Training segmentation models often face three concurrent forms of data insufficiency: scarcity of training scenes, scarcity of point-level annotations, and absence of 2D image sequences from which point clouds were reconstructed.<n>This paper proposes Point pseudo-Label-ing via Open-Vocabulary Image (PLOVIS) model as a pseudo label generator to compensate for the lack of training data.<n>PLOVIS consistently outperforms existing methods including state-of-the-art weakly supervised learning algorithms.
• Score: 0.20305676256390928
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Semantic segmentation of 3D point cloud scenes is a crucial task for various applications. In real-world scenarios, training segmentation models often faces three concurrent forms of data insufficiency: scarcity of training scenes, scarcity of point-level annotations, and absence of 2D image sequences from which point clouds were reconstructed. Existing data-efficient algorithms typically address only one or two of these challenges, leaving the joint treatment of all three unexplored. This paper proposes a data-efficient training framework specifically designed to address the three forms of data insufficiency. Our proposed algorithm, called Point pseudo-Labeling via Open-Vocabulary Image Segmentation (PLOVIS), leverages an Open-Vocabulary Image Segmentation (OVIS) model as a pseudo label generator to compensate for the lack of training data. PLOVIS creates 2D images for pseudo-labeling directly from training 3D point clouds, eliminating the need for 2D image sequences. To mitigate the inherent noise and class imbalance in pseudo labels, we introduce a two-stage filtering of pseudo labels combined with a class-balanced memory bank for effective training. The two-stage filtering mechanism first removes low-confidence pseudo labels, then discards likely incorrect pseudo labels, thereby enhancing the quality of pseudo labels. Experiments on four benchmark datasets, i.e., ScanNet, S3DIS, Toronto3D, and Semantic3D, under realistic data-scarce conditions (a few tens of training 3D scenes, each annotated with only <100 3D points) demonstrate that PLOVIS consistently outperforms existing methods including standard fine-tuning strategies and state-of-the-art weakly supervised learning algorithms. Code will be made publicly available.

Related papers

• Bayesian Self-Training for Semi-Supervised 3D Segmentation [59.544558398992386]
  3D segmentation is a core problem in computer vision. densely labeling 3D point clouds to employ fully-supervised training remains too labor intensive and expensive. Semi-supervised training provides a more practical alternative, where only a small set of labeled data is given, accompanied by a larger unlabeled set.
  arXiv  Detail & Related papers  (2024-09-12T14:54:31Z)
• Towards Modality-agnostic Label-efficient Segmentation with Entropy-Regularized Distribution Alignment [62.73503467108322]
  This topic is widely studied in 3D point cloud segmentation due to the difficulty of annotating point clouds densely. Until recently, pseudo-labels have been widely employed to facilitate training with limited ground-truth labels. Existing pseudo-labeling approaches could suffer heavily from the noises and variations in unlabelled data. We propose a novel learning strategy to regularize the pseudo-labels generated for training, thus effectively narrowing the gaps between pseudo-labels and model predictions.
  arXiv  Detail & Related papers  (2024-08-29T13:31:15Z)
• Label-Efficient 3D Brain Segmentation via Complementary 2D Diffusion Models with Orthogonal Views [10.944692719150071]
  We propose a novel 3D brain segmentation approach using complementary 2D diffusion models. Our goal is to achieve reliable segmentation quality without requiring complete labels for each individual subject.
  arXiv  Detail & Related papers  (2024-07-17T06:14:53Z)
• 3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving [17.42913935045091]
  We propose AFOV, a novel 3D textbfAnnotation-textbfFree framework assisted by 2D textbfOpen-textbfVocabulary segmentation models.<n>It consists of two stages: In the first stage, we innovatively integrate high-quality textual and image features of 2D open-vocabulary models and propose the Tri-Modal contrastive Pre-training (TMP)<n>In the second stage, spatial mapping between point clouds and images is utilized to generate pseudo-labels, enabling cross
  arXiv  Detail & Related papers  (2024-05-24T07:18:09Z)
• U3DS$^3$: Unsupervised 3D Semantic Scene Segmentation [19.706172244951116]
  This paper presents U3DS$3$, as a step towards completely unsupervised point cloud segmentation for any holistic 3D scenes. The initial step of our proposed approach involves generating superpoints based on the geometric characteristics of each scene. We then undergo a learning process through a spatial clustering-based methodology, followed by iterative training using pseudo-labels generated in accordance with the cluster centroids.
  arXiv  Detail & Related papers  (2023-11-10T12:05:35Z)
• Towards Label-free Scene Understanding by Vision Foundation Models [87.13117617056004]
  We investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. We propose a novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths of CLIP and SAM to supervise 2D and 3D networks simultaneously. Our 2D and 3D network achieves label-free semantic segmentation with 28.4% and 33.5% mIoU on ScanNet, improving 4.7% and 7.9%, respectively.
  arXiv  Detail & Related papers  (2023-06-06T17:57:49Z)
• You Only Need One Thing One Click: Self-Training for Weakly Supervised 3D Scene Understanding [107.06117227661204]
  We propose One Thing One Click'', meaning that the annotator only needs to label one point per object. We iteratively conduct the training and label propagation, facilitated by a graph propagation module. Our model can be compatible to 3D instance segmentation equipped with a point-clustering strategy.
  arXiv  Detail & Related papers  (2023-03-26T13:57:00Z)
• CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP [55.864132158596206]
  Contrastive Language-Image Pre-training (CLIP) achieves promising results in 2D zero-shot and few-shot learning. We make the first attempt to investigate how CLIP knowledge benefits 3D scene understanding. We propose CLIP2Scene, a framework that transfers CLIP knowledge from 2D image-text pre-trained models to a 3D point cloud network.
  arXiv  Detail & Related papers  (2023-01-12T10:42:39Z)
• Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning [59.64695628433855]
  We propose a novel cross-modality weakly supervised method for 3D segmentation, incorporating complementary information from unlabeled images. Basically, we design a dual-branch network equipped with an active labeling strategy, to maximize the power of tiny parts of labels. Our method even outperforms the state-of-the-art fully supervised competitors with less than 1% actively selected annotations.
  arXiv  Detail & Related papers  (2022-09-16T07:59:04Z)
• One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation [78.36781565047656]
  We propose "One Thing One Click," meaning that the annotator only needs to label one point per object. We iteratively conduct the training and label propagation, facilitated by a graph propagation module. Our results are also comparable to those of the fully supervised counterparts.
  arXiv  Detail & Related papers  (2021-04-06T02:27:25Z)
• Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes [36.07733308424772]
  The deficiency of 3D segmentation labels is one of the main obstacles to effective point cloud segmentation. We propose a novel deep graph convolutional network-based framework for large-scale semantic scene segmentation in point clouds with sole 2D supervision.
  arXiv  Detail & Related papers  (2020-04-26T23:02:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.