Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation

• URL: http://arxiv.org/abs/2210.06780v1
• Date: Thu, 13 Oct 2022 06:45:07 GMT
• Title: Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation
• Authors: Yuanwei Liu, Nian Liu, Xiwen Yao, Junwei Han
• Abstract summary: Few-shot semantic segmentation aims to segment the target objects in query under the condition of a few annotated support images. We introduce an intermediate prototype for mining both deterministic category information from the support and adaptive category knowledge from the query. In each IPMT layer, we propagate the object information in both support and query features to the prototype and then use it to activate the query feature map.
• Score: 119.51445225693382
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot semantic segmentation aims to segment the target objects in query under the condition of a few annotated support images. Most previous works strive to mine more effective category information from the support to match with the corresponding objects in query. However, they all ignored the category information gap between query and support images. If the objects in them show large intra-class diversity, forcibly migrating the category information from the support to the query is ineffective. To solve this problem, we are the first to introduce an intermediate prototype for mining both deterministic category information from the support and adaptive category knowledge from the query. Specifically, we design an Intermediate Prototype Mining Transformer (IPMT) to learn the prototype in an iterative way. In each IPMT layer, we propagate the object information in both support and query features to the prototype and then use it to activate the query feature map. By conducting this process iteratively, both the intermediate prototype and the query feature can be progressively improved. At last, the final query feature is used to yield precise segmentation prediction. Extensive experiments on both PASCAL-5i and COCO-20i datasets clearly verify the effectiveness of our IPMT and show that it outperforms previous state-of-the-art methods by a large margin. Code is available at https://github.com/LIUYUANWEI98/IPMT

Related papers

• Multi-grained Temporal Prototype Learning for Few-shot Video Object Segmentation [156.4142424784322]
  Few-Shot Video Object (FSVOS) aims to segment objects in a query video with the same category defined by a few annotated support images. We propose to leverage multi-grained temporal guidance information for handling the temporal correlation nature of video data. Our proposed video IPMT model significantly outperforms previous models on two benchmark datasets.
  arXiv  Detail & Related papers  (2023-09-20T09:16:34Z)
• Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation [62.275143240798236]
  Video semantic segmentation dataset has limited categories per video. Less than 10% of queries could be matched to receive meaningful gradient updates during VSS training. Our method achieves state-of-the-art performance on the latest challenging VSS benchmark VSPW without bells and whistles.
  arXiv  Detail & Related papers  (2023-09-14T20:31:06Z)
• MIANet: Aggregating Unbiased Instance and General Information for Few-Shot Semantic Segmentation [6.053853367809978]
  Existing few-shot segmentation methods are based on the meta-learning strategy and extract instance knowledge from a support set. We propose a multi-information aggregation network (MIANet) that effectively leverages the general knowledge, i.e., semantic word embeddings, and instance information for accurate segmentation. Experiments on PASCAL-5i and COCO-20i show that MIANet yields superior performance and set a new state-of-the-art.
  arXiv  Detail & Related papers  (2023-05-23T09:36:27Z)
• Few-shot Medical Image Segmentation via Cross-Reference Transformer [3.2634122554914]
  Few-shot segmentation(FSS) has the potential to address these challenges by learning new categories from a small number of labeled samples. We propose a novel self-supervised few shot medical image segmentation network with Cross-Reference Transformer. Experimental results show that the proposed model achieves good results on both CT dataset and MRI dataset.
  arXiv  Detail & Related papers  (2023-04-19T13:05:18Z)
• Prototype as Query for Few Shot Semantic Segmentation [7.380266341356485]
  Few-shot Semantic (FSS) was proposed to segment unseen classes in a query image, referring to only a few examples named support images. We propose a framework built upon Transformer termed as ProtoFormer to fully capture spatial details in query features.
  arXiv  Detail & Related papers  (2022-11-27T08:41:50Z)
• APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation [56.387647750094466]
  Few-shot semantic segmentation aims to segment novel-class objects in a given query image with only a few labeled support images. Most advanced solutions exploit a metric learning framework that performs segmentation through matching each query feature to a learned class-specific prototype. We present an adaptive prototype representation by introducing class-specific and class-agnostic prototypes.
  arXiv  Detail & Related papers  (2021-11-24T04:38:37Z)
• Dual Prototypical Contrastive Learning for Few-shot Semantic Segmentation [55.339405417090084]
  We propose a dual prototypical contrastive learning approach tailored to the few-shot semantic segmentation (FSS) task. The main idea is to encourage the prototypes more discriminative by increasing inter-class distance while reducing intra-class distance in prototype feature space. We demonstrate that the proposed dual contrastive learning approach outperforms state-of-the-art FSS methods on PASCAL-5i and COCO-20i datasets.
  arXiv  Detail & Related papers  (2021-11-09T08:14:50Z)
• Boosting Few-shot Semantic Segmentation with Transformers [81.43459055197435]
  TRansformer-based Few-shot Semantic segmentation method (TRFS) Our model consists of two modules: Global Enhancement Module (GEM) and Local Enhancement Module (LEM)
  arXiv  Detail & Related papers  (2021-08-04T20:09:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.