Zero-Shot Pseudo Labels Generation Using SAM and CLIP for Semi-Supervised Semantic Segmentation

• URL: http://arxiv.org/abs/2505.19846v2
• Date: Thu, 29 May 2025 08:13:55 GMT
• Title: Zero-Shot Pseudo Labels Generation Using SAM and CLIP for Semi-Supervised Semantic Segmentation
• Authors: Nagito Saito, Shintaro Ito, Koichi Ito, Takafumi Aoki,
• Abstract summary: We propose a method to train a semantic segmentation model using images with annotated labels and pseudo labels.<n>The accuracy of the model depends on the quality of the pseudo labels and the amount of data with annotated labels.<n>The effectiveness of the proposed method is demonstrated through the experiments using the public datasets: PASCAL and MS COCO.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic segmentation is a fundamental task in medical image analysis and autonomous driving and has a problem with the high cost of annotating the labels required in training. To address this problem, semantic segmentation methods based on semi-supervised learning with a small number of labeled data have been proposed. For example, one approach is to train a semantic segmentation model using images with annotated labels and pseudo labels. In this approach, the accuracy of the semantic segmentation model depends on the quality of the pseudo labels, and the quality of the pseudo labels depends on the performance of the model to be trained and the amount of data with annotated labels. In this paper, we generate pseudo labels using zero-shot annotation with the Segment Anything Model (SAM) and Contrastive Language-Image Pretraining (CLIP), improve the accuracy of the pseudo labels using the Unified Dual-Stream Perturbations Approach (UniMatch), and use them as enhanced labels to train a semantic segmentation model. The effectiveness of the proposed method is demonstrated through the experiments using the public datasets: PASCAL and MS COCO. The project web page is available at: https://gsisaoki.github.io/ZERO-SHOT-PLG/

Related papers

• Towards Micro-Action Recognition with Limited Annotations: An Asynchronous Pseudo Labeling and Training Approach [35.32024173141412]
  We introduce the setting of Semi-Supervised MAR (SSMAR), where only a part of samples are labeled.<n>Traditional Semi-Supervised Learning (SSL) methods tend to overfit on inaccurate pseudo-labels, leading to error accumulation and degraded performance.<n>We propose Asynchronous Pseudo Labeling and Training (APLT), which explicitly separates the pseudo-labeling process from model training.
  arXiv  Detail & Related papers  (2025-04-10T14:22:15Z)
• Unpaired Modality Translation for Pseudo Labeling of Histology Images [0.5825410941577593]
  We propose a microscopy pseudo labeling pipeline utilizing unsupervised image translation to address this issue.<n>Our method generates pseudo labels by translating between labeled and unlabeled domains without requiring prior annotation in the target domain.
  arXiv  Detail & Related papers  (2024-12-03T21:45:59Z)
• Scribbles for All: Benchmarking Scribble Supervised Segmentation Across Datasets [51.74296438621836]
  We introduce Scribbles for All, a label and training data generation algorithm for semantic segmentation trained on scribble labels. The main limitation of scribbles as source for weak supervision is the lack of challenging datasets for scribble segmentation. Scribbles for All provides scribble labels for several popular segmentation datasets and provides an algorithm to automatically generate scribble labels for any dataset with dense annotations.
  arXiv  Detail & Related papers  (2024-08-22T15:29:08Z)
• Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation [30.812323329239614]
  Weakly supervised semantic segmentation (WSSS) aims to bypass the need for laborious pixel-level annotation by using only image-level annotation. Most existing methods rely on Class Activation Maps (CAM) to derive pixel-level pseudo-labels. We introduce a simple yet effective method harnessing the Segment Anything Model (SAM), a class-agnostic foundation model capable of producing fine-grained instance masks of objects, parts, and subparts.
  arXiv  Detail & Related papers  (2023-05-09T23:24:09Z)
• LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds [62.49198183539889]
  We propose a label-efficient semantic segmentation pipeline for outdoor scenes with LiDAR point clouds. Our method co-designs an efficient labeling process with semi/weakly supervised learning. Our proposed method is even highly competitive compared to the fully supervised counterpart with 100% labels.
  arXiv  Detail & Related papers  (2022-10-14T19:13:36Z)
• Scaling up Multi-domain Semantic Segmentation with Sentence Embeddings [81.09026586111811]
  We propose an approach to semantic segmentation that achieves state-of-the-art supervised performance when applied in a zero-shot setting. This is achieved by replacing each class label with a vector-valued embedding of a short paragraph that describes the class. The resulting merged semantic segmentation dataset of over 2 Million images enables training a model that achieves performance equal to that of state-of-the-art supervised methods on 7 benchmark datasets.
  arXiv  Detail & Related papers  (2022-02-04T07:19:09Z)
• GuidedMix-Net: Semi-supervised Semantic Segmentation by Using Labeled Images as Reference [90.5402652758316]
  We propose a novel method for semi-supervised semantic segmentation named GuidedMix-Net. It uses labeled information to guide the learning of unlabeled instances. It achieves competitive segmentation accuracy and significantly improves the mIoU by +7$%$ compared to previous approaches.
  arXiv  Detail & Related papers  (2021-12-28T06:48:03Z)
• Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation [30.688753736660725]
  We present a simple and yet effective Distribution Alignment and Random Sampling (DARS) method to produce unbiased pseudo labels. Our method performs favorably in comparison with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-07-23T14:45:14Z)
• A Closer Look at Self-training for Zero-Label Semantic Segmentation [53.4488444382874]
  Being able to segment unseen classes not observed during training is an important technical challenge in deep learning. Prior zero-label semantic segmentation works approach this task by learning visual-semantic embeddings or generative models. We propose a consistency regularizer to filter out noisy pseudo-labels by taking the intersections of the pseudo-labels generated from different augmentations of the same image.
  arXiv  Detail & Related papers  (2021-04-21T14:34:33Z)
• One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation [78.36781565047656]
  We propose "One Thing One Click," meaning that the annotator only needs to label one point per object. We iteratively conduct the training and label propagation, facilitated by a graph propagation module. Our results are also comparable to those of the fully supervised counterparts.
  arXiv  Detail & Related papers  (2021-04-06T02:27:25Z)
• PseudoSeg: Designing Pseudo Labels for Semantic Segmentation [78.35515004654553]
  We present a re-design of pseudo-labeling to generate structured pseudo labels for training with unlabeled or weakly-labeled data. We demonstrate the effectiveness of the proposed pseudo-labeling strategy in both low-data and high-data regimes.
  arXiv  Detail & Related papers  (2020-10-19T17:59:30Z)
• PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervision [12.284208932393073]
  This paper presents a novel procedure for producing semantic segmentation from images given some point level annotations. We propose training a CNN that is normally fully supervised using our pseudo labels in place of ground truth labels. Our method achieves state of the art results for point supervised semantic segmentation on the PASCAL VOC 2012 dataset citeeveringham2010pascal, even outperforming state of the art methods for stronger bounding box and squiggle supervision.
  arXiv  Detail & Related papers  (2020-07-10T21:25:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.