Rethinking Prior Information Generation with CLIP for Few-Shot Segmentation

• URL: http://arxiv.org/abs/2405.08458v1
• Date: Tue, 14 May 2024 09:28:25 GMT
• Title: Rethinking Prior Information Generation with CLIP for Few-Shot Segmentation
• Authors: Jin Wang, Bingfeng Zhang, Jian Pang, Honglong Chen, Weifeng Liu,
• Abstract summary: We propose to replace the visual prior representation with the visual-text alignment capacity to capture more reliable guidance. We show that our method obtains a clearly substantial improvement and reaches the new state-of-the-art performance.
• Score: 14.998239253285394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot segmentation remains challenging due to the limitations of its labeling information for unseen classes. Most previous approaches rely on extracting high-level feature maps from the frozen visual encoder to compute the pixel-wise similarity as a key prior guidance for the decoder. However, such a prior representation suffers from coarse granularity and poor generalization to new classes since these high-level feature maps have obvious category bias. In this work, we propose to replace the visual prior representation with the visual-text alignment capacity to capture more reliable guidance and enhance the model generalization. Specifically, we design two kinds of training-free prior information generation strategy that attempts to utilize the semantic alignment capability of the Contrastive Language-Image Pre-training model (CLIP) to locate the target class. Besides, to acquire more accurate prior guidance, we build a high-order relationship of attention maps and utilize it to refine the initial prior information. Experiments on both the PASCAL-5{i} and COCO-20{i} datasets show that our method obtains a clearly substantial improvement and reaches the new state-of-the-art performance.

Related papers

• PEPL: Precision-Enhanced Pseudo-Labeling for Fine-Grained Image Classification in Semi-Supervised Learning [3.801446153948012]
  We introduce Precision-Enhanced Pseudo-Labeling(PEPL) approach for fine-grained image classification within a semi-supervised learning framework. Our method leverages the abundance of unlabeled data by generating high-quality pseudo-labels. We achieve state-of-the-art performance on benchmark datasets, demonstrating significant improvements over existing semi-supervised strategies.
  arXiv  Detail & Related papers  (2024-09-05T02:32:07Z)
• Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification [62.425462136772666]
  Fine-grained ship classification in remote sensing (RS-FGSC) poses a significant challenge due to the high similarity between classes and the limited availability of labeled data. Recent advancements in large pre-trained Vision-Language Models (VLMs) have demonstrated impressive capabilities in few-shot or zero-shot learning. This study delves into harnessing the potential of VLMs to enhance classification accuracy for unseen ship categories.
  arXiv  Detail & Related papers  (2024-03-13T05:48:58Z)
• Gramian Attention Heads are Strong yet Efficient Vision Learners [26.79263390835444]
  We introduce a novel architecture design that enhances expressiveness by incorporating multiple head classifiers (ie, classification heads) Our approach employs attention-based aggregation, utilizing pairwise feature similarity to enhance multiple lightweight heads with minimal resource overhead. Our models eventually surpass state-of-the-art CNNs and ViTs regarding the accuracy-grained trade-off on ImageNet-1K.
  arXiv  Detail & Related papers  (2023-10-25T09:08:58Z)
• CLIP Brings Better Features to Visual Aesthetics Learners [12.0962117940694]
  Image aesthetics assessment (IAA) is one of the ideal application scenarios for such methods due to subjective and expensive labeling procedure. In this work, an unified and flexible two-phase textbfCLIP-based textbfSemi-supervised textbfKnowledge textbfDistillation paradigm is proposed, namely textbftextitCSKD.
  arXiv  Detail & Related papers  (2023-07-28T16:00:21Z)
• Learning What Not to Segment: A New Perspective on Few-Shot Segmentation [63.910211095033596]
  Recently few-shot segmentation (FSS) has been extensively developed. This paper proposes a fresh and straightforward insight to alleviate the problem. In light of the unique nature of the proposed approach, we also extend it to a more realistic but challenging setting.
  arXiv  Detail & Related papers  (2022-03-15T03:08:27Z)
• Modeling the Background for Incremental and Weakly-Supervised Semantic Segmentation [39.025848280224785]
  We introduce a novel incremental class learning approach for semantic segmentation. Since each training step provides annotation only for a subset of all possible classes, pixels of the background class exhibit a semantic shift. We demonstrate the effectiveness of our approach with an extensive evaluation on the Pascal-VOC, ADE20K, and Cityscapes datasets.
  arXiv  Detail & Related papers  (2022-01-31T16:33:21Z)
• A Simple Baseline for Zero-shot Semantic Segmentation with Pre-trained Vision-language Model [61.58071099082296]
  It is unclear how to make zero-shot recognition working well on broader vision problems, such as object detection and semantic segmentation. In this paper, we target for zero-shot semantic segmentation, by building it on an off-the-shelf pre-trained vision-language model, i.e., CLIP. Our experimental results show that this simple framework surpasses previous state-of-the-arts by a large margin.
  arXiv  Detail & Related papers  (2021-12-29T18:56:18Z)
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting [91.56988987393483]
  We present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones.
  arXiv  Detail & Related papers  (2021-12-02T18:59:32Z)
• Semantic-Aware Generation for Self-Supervised Visual Representation Learning [116.5814634936371]
  We advocate for Semantic-aware Generation (SaGe) to facilitate richer semantics rather than details to be preserved in the generated image. SaGe complements the target network with view-specific features and thus alleviates the semantic degradation brought by intensive data augmentations. We execute SaGe on ImageNet-1K and evaluate the pre-trained models on five downstream tasks including nearest neighbor test, linear classification, and fine-scaled image recognition.
  arXiv  Detail & Related papers  (2021-11-25T16:46:13Z)
• Calibrating Class Activation Maps for Long-Tailed Visual Recognition [60.77124328049557]
  We present two effective modifications of CNNs to improve network learning from long-tailed distribution. First, we present a Class Activation Map (CAMC) module to improve the learning and prediction of network classifiers. Second, we investigate the use of normalized classifiers for representation learning in long-tailed problems.
  arXiv  Detail & Related papers  (2021-08-29T05:45:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.