Foreground-Covering Prototype Generation and Matching for SAM-Aided Few-Shot Segmentation

• URL: http://arxiv.org/abs/2501.00752v1
• Date: Wed, 01 Jan 2025 06:43:18 GMT
• Title: Foreground-Covering Prototype Generation and Matching for SAM-Aided Few-Shot Segmentation
• Authors: Suho Park, SuBeen Lee, Hyun Seok Seong, Jaejoon Yoo, Jae-Pil Heo,
• Abstract summary: We propose Foreground-Covering Prototype Generation and Matching to resolve Few-Shot (FSS) We construct support and query prototypes with SAM features and distinguish query prototypes of target regions based on ResNet features. Our state-of-the-art performances on various datasets validate the effectiveness of the proposed method for FSS.
• Score: 15.301024379147709
• License:
• Abstract: We propose Foreground-Covering Prototype Generation and Matching to resolve Few-Shot Segmentation (FSS), which aims to segment target regions in unlabeled query images based on labeled support images. Unlike previous research, which typically estimates target regions in the query using support prototypes and query pixels, we utilize the relationship between support and query prototypes. To achieve this, we utilize two complementary features: SAM Image Encoder features for pixel aggregation and ResNet features for class consistency. Specifically, we construct support and query prototypes with SAM features and distinguish query prototypes of target regions based on ResNet features. For the query prototype construction, we begin by roughly guiding foreground regions within SAM features using the conventional pseudo-mask, then employ iterative cross-attention to aggregate foreground features into learnable tokens. Here, we discover that the cross-attention weights can effectively alternate the conventional pseudo-mask. Therefore, we use the attention-based pseudo-mask to guide ResNet features to focus on the foreground, then infuse the guided ResNet feature into the learnable tokens to generate class-consistent query prototypes. The generation of the support prototype is conducted symmetrically to that of the query one, with the pseudo-mask replaced by the ground-truth mask. Finally, we compare these query prototypes with support ones to generate prompts, which subsequently produce object masks through the SAM Mask Decoder. Our state-of-the-art performances on various datasets validate the effectiveness of the proposed method for FSS. Our official code is available at https://github.com/SuhoPark0706/FCP

Related papers

• Bridge the Points: Graph-based Few-shot Segment Anything Semantically [79.1519244940518]
  Recent advancements in pre-training techniques have enhanced the capabilities of vision foundation models. Recent studies extend the SAM to Few-shot Semantic segmentation (FSS) We propose a simple yet effective approach based on graph analysis.
  arXiv  Detail & Related papers  (2024-10-09T15:02:28Z)
• Correlation Weighted Prototype-based Self-Supervised One-Shot Segmentation of Medical Images [12.365801596593936]
  Medical image segmentation is one of the domains where sufficient annotated data is not available. We propose a prototype-based self-supervised one-way one-shot learning framework using pseudo-labels generated from superpixels. We show that the proposed simple but potent framework performs at par with the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-12T15:38:51Z)
• Support-Query Prototype Fusion Network for Few-shot Medical Image Segmentation [7.6695642174485705]
  Few-shot learning, which utilizes a small amount of labeled data to generalize to unseen classes, has emerged as a critical research area. We propose a novel Support-Query Prototype Fusion Network (SQPFNet) to mitigate this drawback. evaluation results on two public datasets, SABS and CMR, show that SQPFNet achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-05-13T07:31:16Z)
• Self-Regularized Prototypical Network for Few-Shot Semantic Segmentation [31.445316481839335]
  We tackle the few-shot segmentation using a self-regularized network (SRPNet) based on prototype extraction for better utilization of the support information. A direct yet effective prototype regularization on support set is proposed in SRPNet, in which the generated prototypes are evaluated and regularized on the support set itself. Our proposed SRPNet achieves new state-of-art performance on 1-shot and 5-shot segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-10-30T12:43:07Z)
• Dynamic Prototype Mask for Occluded Person Re-Identification [88.7782299372656]
  Existing methods mainly address this issue by employing body clues provided by an extra network to distinguish the visible part. We propose a novel Dynamic Prototype Mask (DPM) based on two self-evident prior knowledge. Under this condition, the occluded representation could be well aligned in a selected subspace spontaneously.
  arXiv  Detail & Related papers  (2022-07-19T03:31:13Z)
• Dynamic Focus-aware Positional Queries for Semantic Segmentation [94.6834904076914]
  We propose a simple yet effective query design for semantic segmentation termed Dynamic Focus-aware Positional Queries. Our framework achieves SOTA performance and outperforms Mask2former by clear margins of 1.1%, 1.9%, and 1.1% single-scale mIoU with ResNet-50, Swin-T, and Swin-B backbones.
  arXiv  Detail & Related papers  (2022-04-04T05:16:41Z)
• Semantically Meaningful Class Prototype Learning for One-Shot Image Semantic Segmentation [58.96902899546075]
  One-shot semantic image segmentation aims to segment the object regions for the novel class with only one annotated image. Recent works adopt the episodic training strategy to mimic the expected situation at testing time. We propose to leverage the multi-class label information during the episodic training. It will encourage the network to generate more semantically meaningful features for each category.
  arXiv  Detail & Related papers  (2021-02-22T12:07:35Z)
• Prototype Mixture Models for Few-shot Semantic Segmentation [50.866870384596446]
  Few-shot segmentation is challenging because objects within the support and query images could significantly differ in appearance and pose. We propose prototype mixture models (PMMs), which correlate diverse image regions with multiple prototypes to enforce the prototype-based semantic representation. PMMs improve 5-shot segmentation performance on MS-COCO by up to 5.82% with only a moderate cost for model size and inference speed.
  arXiv  Detail & Related papers  (2020-08-10T04:33:17Z)
• PointINS: Point-based Instance Segmentation [117.38579097923052]
  Mask representation in instance segmentation with Point-of-Interest (PoI) features is challenging because learning a high-dimensional mask feature for each instance requires a heavy computing burden. We propose an instance-aware convolution, which decomposes this mask representation learning task into two tractable modules. Along with instance-aware convolution, we propose PointINS, a simple and practical instance segmentation approach.
  arXiv  Detail & Related papers  (2020-03-13T08:24:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.