Iterative Few-shot Semantic Segmentation from Image Label Text

• URL: http://arxiv.org/abs/2303.05646v1
• Date: Fri, 10 Mar 2023 01:48:14 GMT
• Title: Iterative Few-shot Semantic Segmentation from Image Label Text
• Authors: Haohan Wang, Liang Liu, Wuhao Zhang, Jiangning Zhang, Zhenye Gan, Yabiao Wang, Chengjie Wang, Haoqian Wang
• Abstract summary: Few-shot semantic segmentation aims to learn to segment unseen class objects with the guidance of only a few support images. We propose a general framework to generate coarse masks with the help of the powerful vision-language model CLIP. Our method owns an excellent generalization ability for the images in the wild and uncommon classes.
• Score: 36.53926941601841
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot semantic segmentation aims to learn to segment unseen class objects with the guidance of only a few support images. Most previous methods rely on the pixel-level label of support images. In this paper, we focus on a more challenging setting, in which only the image-level labels are available. We propose a general framework to firstly generate coarse masks with the help of the powerful vision-language model CLIP, and then iteratively and mutually refine the mask predictions of support and query images. Extensive experiments on PASCAL-5i and COCO-20i datasets demonstrate that our method not only outperforms the state-of-the-art weakly supervised approaches by a significant margin, but also achieves comparable or better results to recent supervised methods. Moreover, our method owns an excellent generalization ability for the images in the wild and uncommon classes. Code will be available at https://github.com/Whileherham/IMR-HSNet.

Related papers

• Learning Semantic Segmentation with Query Points Supervision on Aerial Images [57.09251327650334]
  We present a weakly supervised learning algorithm to train semantic segmentation algorithms. Our proposed approach performs accurate semantic segmentation and improves efficiency by significantly reducing the cost and time required for manual annotation.
  arXiv  Detail & Related papers  (2023-09-11T14:32:04Z)
• Masked Cross-image Encoding for Few-shot Segmentation [16.445813548503708]
  Few-shot segmentation (FSS) is a dense prediction task that aims to infer the pixel-wise labels of unseen classes using only a limited number of annotated images. We propose a joint learning method termed Masked Cross-Image MCE, which is designed to capture common visual properties that describe object details and to learn bidirectional inter-image dependencies that enhance feature interaction.
  arXiv  Detail & Related papers  (2023-08-22T05:36:39Z)
• ISLE: A Framework for Image Level Semantic Segmentation Ensemble [5.137284292672375]
  Conventional semantic segmentation networks require massive pixel-wise annotated labels to reach state-of-the-art prediction quality. We propose ISLE, which employs an ensemble of the "pseudo-labels" for a given set of different semantic segmentation techniques on a class-wise level. We reach up to 2.4% improvement over ISLE's individual components.
  arXiv  Detail & Related papers  (2023-03-14T13:36:36Z)
• A Simple Baseline for Zero-shot Semantic Segmentation with Pre-trained Vision-language Model [61.58071099082296]
  It is unclear how to make zero-shot recognition working well on broader vision problems, such as object detection and semantic segmentation. In this paper, we target for zero-shot semantic segmentation, by building it on an off-the-shelf pre-trained vision-language model, i.e., CLIP. Our experimental results show that this simple framework surpasses previous state-of-the-arts by a large margin.
  arXiv  Detail & Related papers  (2021-12-29T18:56:18Z)
• Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples [34.97652735163338]
  We propose a novel approach for creating semantic segmentation masks for every object. Our method takes as input the image-level labels of the class categories present in the image. The output of this stage provides pixel-level pseudo-labels, instead of the manual pixel-level labels required by supervised methods.
  arXiv  Detail & Related papers  (2021-12-06T17:32:38Z)
• A Pixel-Level Meta-Learner for Weakly Supervised Few-Shot Semantic Segmentation [40.27705176115985]
  Few-shot semantic segmentation addresses the learning task in which only few images with ground truth pixel-level labels are available for the novel classes of interest. We propose a novel meta-learning framework, which predicts pseudo pixel-level segmentation masks from a limited amount of data and their semantic labels. Our proposed learning model can be viewed as a pixel-level meta-learner.
  arXiv  Detail & Related papers  (2021-11-02T08:28:11Z)
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization [112.68171734288237]
  We propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. We learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization.
  arXiv  Detail & Related papers  (2021-04-12T21:41:25Z)
• Semantically Meaningful Class Prototype Learning for One-Shot Image Semantic Segmentation [58.96902899546075]
  One-shot semantic image segmentation aims to segment the object regions for the novel class with only one annotated image. Recent works adopt the episodic training strategy to mimic the expected situation at testing time. We propose to leverage the multi-class label information during the episodic training. It will encourage the network to generate more semantically meaningful features for each category.
  arXiv  Detail & Related papers  (2021-02-22T12:07:35Z)
• Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals [78.12377360145078]
  We introduce a novel two-step framework that adopts a predetermined prior in a contrastive optimization objective to learn pixel embeddings. This marks a large deviation from existing works that relied on proxy tasks or end-to-end clustering. In particular, when fine-tuning the learned representations using just 1% of labeled examples on PASCAL, we outperform supervised ImageNet pre-training by 7.1% mIoU.
  arXiv  Detail & Related papers  (2021-02-11T18:54:47Z)
• Grafit: Learning fine-grained image representations with coarse labels [114.17782143848315]
  This paper tackles the problem of learning a finer representation than the one provided by training labels. By jointly leveraging the coarse labels and the underlying fine-grained latent space, it significantly improves the accuracy of category-level retrieval methods.
  arXiv  Detail & Related papers  (2020-11-25T19:06:26Z)
• Few-Shot Semantic Segmentation Augmented with Image-Level Weak Annotations [23.02986307143718]
  Recent progress in fewshot semantic segmentation tackles the issue by only a few pixel-level annotated examples. Our key idea is to learn a better prototype representation of the class by fusing the knowledge from the image-level labeled data. We propose a new framework, called PAIA, to learn the class prototype representation in a metric space by integrating image-level annotations.
  arXiv  Detail & Related papers  (2020-07-03T04:58:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.