Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation

• URL: http://arxiv.org/abs/2302.14250v2
• Date: Thu, 20 Apr 2023 08:12:44 GMT
• Title: Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation
• Authors: Chaohui Yu, Qiang Zhou, Jingliang Li, Jianlong Yuan, Zhibin Wang, Fan Wang
• Abstract summary: Weakly incremental learning for semantic segmentation (WILSS) is a novel and attractive task. We propose a novel and data-efficient framework for WILSS, named FMWISS.
• Score: 12.362400851574872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern incremental learning for semantic segmentation methods usually learn new categories based on dense annotations. Although achieve promising results, pixel-by-pixel labeling is costly and time-consuming. Weakly incremental learning for semantic segmentation (WILSS) is a novel and attractive task, which aims at learning to segment new classes from cheap and widely available image-level labels. Despite the comparable results, the image-level labels can not provide details to locate each segment, which limits the performance of WILSS. This inspires us to think how to improve and effectively utilize the supervision of new classes given image-level labels while avoiding forgetting old ones. In this work, we propose a novel and data-efficient framework for WILSS, named FMWISS. Specifically, we propose pre-training based co-segmentation to distill the knowledge of complementary foundation models for generating dense pseudo labels. We further optimize the noisy pseudo masks with a teacher-student architecture, where a plug-in teacher is optimized with a proposed dense contrastive loss. Moreover, we introduce memory-based copy-paste augmentation to improve the catastrophic forgetting problem of old classes. Extensive experiments on Pascal VOC and COCO datasets demonstrate the superior performance of our framework, e.g., FMWISS achieves 70.7% and 73.3% in the 15-5 VOC setting, outperforming the state-of-the-art method by 3.4% and 6.1%, respectively.

Related papers

• A Self Supervised StyleGAN for Image Annotation and Classification with Extremely Limited Labels [35.43549147657739]
  We propose SS-StyleGAN, a self-supervised approach for image annotation and classification suitable for extremely small annotated datasets. We show that the proposed method attains strong classification results using small labeled datasets of sizes 50 and even 10.
  arXiv  Detail & Related papers  (2023-12-26T09:46:50Z)
• Improving Semi-Supervised Semantic Segmentation with Dual-Level Siamese Structure Network [7.438140196173472]
  We propose a dual-level Siamese structure network (DSSN) for pixel-wise contrastive learning. We introduce a novel class-aware pseudo-label selection strategy for weak-to-strong supervision. Our proposed method achieves state-of-the-art results on two datasets.
  arXiv  Detail & Related papers  (2023-07-26T03:30:28Z)
• Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation [57.80416375466496]
  Incremental or continual learning has been extensively studied for image classification tasks to alleviate catastrophic forgetting. We propose a simple yet effective method in this paper, named unseen Classes via Regional Objectness for Mining (MicroSeg) Our MicroSeg is based on the assumption that background regions with strong objectness possibly belong to those concepts in the historical or future stages. In this way, the distribution characterizes of old concepts in the feature space could be better perceived, relieving the catastrophic forgetting caused by the background shift accordingly.
  arXiv  Detail & Related papers  (2022-11-13T10:06:17Z)
• Novel Class Discovery in Semantic Segmentation [104.30729847367104]
  We introduce a new setting of Novel Class Discovery in Semantic (NCDSS) It aims at segmenting unlabeled images containing new classes given prior knowledge from a labeled set of disjoint classes. In NCDSS, we need to distinguish the objects and background, and to handle the existence of multiple classes within an image. We propose the Entropy-based Uncertainty Modeling and Self-training (EUMS) framework to overcome noisy pseudo-labels.
  arXiv  Detail & Related papers  (2021-12-03T13:31:59Z)
• Incremental Learning in Semantic Segmentation from Image Labels [18.404068463921426]
  Existing semantic segmentation approaches achieve impressive results, but struggle to update their models incrementally as new categories are uncovered. This paper proposes a novel framework for Weakly Incremental Learning for Semantics, that aims at learning to segment new classes from cheap and largely available image-level labels. As opposed to existing approaches, that need to generate pseudo-labels offline, we use an auxiliary classifier, trained with image-level labels and regularized by the segmentation model, to obtain pseudo-supervision online and update the model incrementally.
  arXiv  Detail & Related papers  (2021-12-03T12:47:12Z)
• A Pixel-Level Meta-Learner for Weakly Supervised Few-Shot Semantic Segmentation [40.27705176115985]
  Few-shot semantic segmentation addresses the learning task in which only few images with ground truth pixel-level labels are available for the novel classes of interest. We propose a novel meta-learning framework, which predicts pseudo pixel-level segmentation masks from a limited amount of data and their semantic labels. Our proposed learning model can be viewed as a pixel-level meta-learner.
  arXiv  Detail & Related papers  (2021-11-02T08:28:11Z)
• Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation [88.49669148290306]
  We propose a novel weakly supervised multi-task framework called AuxSegNet to leverage saliency detection and multi-label image classification as auxiliary tasks. Inspired by their similar structured semantics, we also propose to learn a cross-task global pixel-level affinity map from the saliency and segmentation representations. The learned cross-task affinity can be used to refine saliency predictions and propagate CAM maps to provide improved pseudo labels for both tasks.
  arXiv  Detail & Related papers  (2021-07-25T11:39:58Z)
• A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation [74.8791451327354]
  We propose a simple yet effective semi-supervised learning framework for semantic segmentation. A set of simple design and training techniques can collectively improve the performance of semi-supervised semantic segmentation significantly. Our method achieves state-of-the-art results in the semi-supervised settings on the Cityscapes and Pascal VOC datasets.
  arXiv  Detail & Related papers  (2021-04-15T06:01:39Z)
• A Few Guidelines for Incremental Few-Shot Segmentation [57.34237650765928]
  Given a pretrained segmentation model and few images containing novel classes, our goal is to learn to segment novel classes while retaining the ability to segment previously seen ones. We show how the main problems of end-to-end training in this scenario are. i) the drift of the batch-normalization statistics toward novel classes that we can fix with batch renormalization and. ii) the forgetting of old classes, that we can fix with regularization strategies.
  arXiv  Detail & Related papers  (2020-11-30T20:45:56Z)
• Grafit: Learning fine-grained image representations with coarse labels [114.17782143848315]
  This paper tackles the problem of learning a finer representation than the one provided by training labels. By jointly leveraging the coarse labels and the underlying fine-grained latent space, it significantly improves the accuracy of category-level retrieval methods.
  arXiv  Detail & Related papers  (2020-11-25T19:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.