Generalized Category Discovery in Semantic Segmentation

• URL: http://arxiv.org/abs/2311.11525v1
• Date: Mon, 20 Nov 2023 04:11:16 GMT
• Title: Generalized Category Discovery in Semantic Segmentation
• Authors: Zhengyuan Peng, Qijian Tian, Jianqing Xu, Yizhang Jin, Xuequan Lu, Xin Tan, Yuan Xie, Lizhuang Ma
• Abstract summary: This paper explores a novel setting called Generalized Category Discovery in Semantic (GCDSS) GCDSS aims to segment unlabeled images given prior knowledge from a labeled set of base classes. In contrast to Novel Category Discovery in Semantic (NCDSS), there is no prerequisite for prior knowledge mandating the existence of at least one novel class in each unlabeled image.
• Score: 43.99230778597973
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper explores a novel setting called Generalized Category Discovery in Semantic Segmentation (GCDSS), aiming to segment unlabeled images given prior knowledge from a labeled set of base classes. The unlabeled images contain pixels of the base class or novel class. In contrast to Novel Category Discovery in Semantic Segmentation (NCDSS), there is no prerequisite for prior knowledge mandating the existence of at least one novel class in each unlabeled image. Besides, we broaden the segmentation scope beyond foreground objects to include the entire image. Existing NCDSS methods rely on the aforementioned priors, making them challenging to truly apply in real-world situations. We propose a straightforward yet effective framework that reinterprets the GCDSS challenge as a task of mask classification. Additionally, we construct a baseline method and introduce the Neighborhood Relations-Guided Mask Clustering Algorithm (NeRG-MaskCA) for mask categorization to address the fragmentation in semantic representation. A benchmark dataset, Cityscapes-GCD, derived from the Cityscapes dataset, is established to evaluate the GCDSS framework. Our method demonstrates the feasibility of the GCDSS problem and the potential for discovering and segmenting novel object classes in unlabeled images. We employ the generated pseudo-labels from our approach as ground truth to supervise the training of other models, thereby enabling them with the ability to segment novel classes. It paves the way for further research in generalized category discovery, broadening the horizons of semantic segmentation and its applications. For details, please visit https://github.com/JethroPeng/GCDSS

Related papers

• Vocabulary-free Image Classification and Semantic Segmentation [71.78089106671581]
  We introduce the Vocabulary-free Image Classification (VIC) task, which aims to assign a class from an un-constrained language-induced semantic space to an input image without needing a known vocabulary. VIC is challenging due to the vastness of the semantic space, which contains millions of concepts, including fine-grained categories. We propose Category Search from External Databases (CaSED), a training-free method that leverages a pre-trained vision-language model and an external database.
  arXiv  Detail & Related papers  (2024-04-16T19:27:21Z)
• Active Generalized Category Discovery [60.69060965936214]
  Generalized Category Discovery (GCD) endeavors to cluster unlabeled samples from both novel and old classes. We take the spirit of active learning and propose a new setting called Active Generalized Category Discovery (AGCD) Our method achieves state-of-the-art performance on both generic and fine-grained datasets.
  arXiv  Detail & Related papers  (2024-03-07T07:12:24Z)
• Unsupervised Universal Image Segmentation [59.0383635597103]
  We propose an Unsupervised Universal model (U2Seg) adept at performing various image segmentation tasks. U2Seg generates pseudo semantic labels for these segmentation tasks via leveraging self-supervised models. We then self-train the model on these pseudo semantic labels, yielding substantial performance gains.
  arXiv  Detail & Related papers  (2023-12-28T18:59:04Z)
• Learning Semi-supervised Gaussian Mixture Models for Generalized Category Discovery [36.01459228175808]
  We propose an EM-like framework that alternates between representation learning and class number estimation. We evaluate our framework on both generic image classification datasets and challenging fine-grained object recognition datasets.
  arXiv  Detail & Related papers  (2023-05-10T13:47:38Z)
• Weak-shot Semantic Segmentation via Dual Similarity Transfer [33.18870478560099]
  We propose SimFormer, which performs dual similarity transfer upon MaskFormer. Proposal segmentation allows proposal-pixel similarity transfer from base classes to novel classes. We also learn pixel-pixel similarity from base classes and distill such class-agnostic semantic similarity to the semantic masks of novel classes.
  arXiv  Detail & Related papers  (2022-10-05T13:54:34Z)
• Segmenting Known Objects and Unseen Unknowns without Prior Knowledge [86.46204148650328]
  holistic segmentation aims to identify and separate objects of unseen, unknown categories into instances without any prior knowledge about them. We tackle this new problem with U3HS, which finds unknowns as highly uncertain regions and clusters their corresponding instance-aware embeddings into individual objects. Experiments on public data from MS, Cityscapes, and Lost&Found demonstrate the effectiveness of U3HS.
  arXiv  Detail & Related papers  (2022-09-12T16:59:36Z)
• GOSS: Towards Generalized Open-set Semantic Segmentation [38.00665104750745]
  We present and study a new image segmentation task, called Generalized Open-set Semantic (GOSS) GOSS classifies pixels as belonging to known classes, and clusters (or groups) of pixels of unknown class are labelled as such. We believe our new GOSS task can produce an expressive image understanding for future research.
  arXiv  Detail & Related papers  (2022-03-23T01:03:06Z)
• Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples [34.97652735163338]
  We propose a novel approach for creating semantic segmentation masks for every object. Our method takes as input the image-level labels of the class categories present in the image. The output of this stage provides pixel-level pseudo-labels, instead of the manual pixel-level labels required by supervised methods.
  arXiv  Detail & Related papers  (2021-12-06T17:32:38Z)
• Novel Class Discovery in Semantic Segmentation [104.30729847367104]
  We introduce a new setting of Novel Class Discovery in Semantic (NCDSS) It aims at segmenting unlabeled images containing new classes given prior knowledge from a labeled set of disjoint classes. In NCDSS, we need to distinguish the objects and background, and to handle the existence of multiple classes within an image. We propose the Entropy-based Uncertainty Modeling and Self-training (EUMS) framework to overcome noisy pseudo-labels.
  arXiv  Detail & Related papers  (2021-12-03T13:31:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.