Learning to Adapt Category Consistent Meta-Feature of CLIP for Few-Shot Classification

• URL: http://arxiv.org/abs/2407.05647v1
• Date: Mon, 8 Jul 2024 06:18:04 GMT
• Title: Learning to Adapt Category Consistent Meta-Feature of CLIP for Few-Shot Classification
• Authors: Jiaying Shi, Xuetong Xue, Shenghui Xu,
• Abstract summary: Recent CLIP-based methods have shown promising zero-shot and few-shot performance on image classification tasks. We propose the Meta-Feature Adaption method (MF-Adapter) that combines the complementary strengths of both LRs and high-level semantic representations. Our proposed method is superior to the state-of-the-art CLIP downstream few-shot classification methods, even showing stronger performance on a set of challenging visual classification tasks.
• Score: 1.6385815610837167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent CLIP-based methods have shown promising zero-shot and few-shot performance on image classification tasks. Existing approaches such as CoOp and Tip-Adapter only focus on high-level visual features that are fully aligned with textual features representing the ``Summary" of the image. However, the goal of few-shot learning is to classify unseen images of the same category with few labeled samples. Especially, in contrast to high-level representations, local representations (LRs) at low-level are more consistent between seen and unseen samples. Based on this point, we propose the Meta-Feature Adaption method (MF-Adapter) that combines the complementary strengths of both LRs and high-level semantic representations. Specifically, we introduce the Meta-Feature Unit (MF-Unit), which is a simple yet effective local similarity metric to measure category-consistent local context in an inductive manner. Then we train an MF-Adapter to map image features to MF-Unit for adequately generalizing the intra-class knowledge between unseen images and the support set. Extensive experiments show that our proposed method is superior to the state-of-the-art CLIP downstream few-shot classification methods, even showing stronger performance on a set of challenging visual classification tasks.

Related papers

• Beyond Mask: Rethinking Guidance Types in Few-shot Segmentation [67.35274834837064]
  We develop a universal vision-language framework (UniFSS) to integrate prompts from text, mask, box, and image. UniFSS significantly outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-16T08:41:01Z)
• Not All Instances Contribute Equally: Instance-adaptive Class Representation Learning for Few-Shot Visual Recognition [94.04041301504567]
  Few-shot visual recognition refers to recognize novel visual concepts from a few labeled instances. We propose a novel metric-based meta-learning framework termed instance-adaptive class representation learning network (ICRL-Net) for few-shot visual recognition.
  arXiv  Detail & Related papers  (2022-09-07T10:00:18Z)
• Matching Feature Sets for Few-Shot Image Classification [22.84472344406448]
  We argue that a set-based representation intrinsically builds a richer representation of images from the base classes. Our approach, dubbed SetFeat, embeds shallow self-attention mechanisms inside existing encoder architectures.
  arXiv  Detail & Related papers  (2022-04-02T22:42:54Z)
• CAD: Co-Adapting Discriminative Features for Improved Few-Shot Classification [11.894289991529496]
  Few-shot classification is a challenging problem that aims to learn a model that can adapt to unseen classes given a few labeled samples. Recent approaches pre-train a feature extractor, and then fine-tune for episodic meta-learning. We propose a strategy to cross-attend and re-weight discriminative features for few-shot classification.
  arXiv  Detail & Related papers  (2022-03-25T06:14:51Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)
• Few-shot Image Classification with Multi-Facet Prototypes [48.583388368897126]
  We organize visual features into facets, which intuitively group features of the same kind. It is possible to predict facet importance from a pre-trained embedding of the category names. In particular, we propose an adaptive similarity measure, relying on predicted facet importance weights for a given set of categories.
  arXiv  Detail & Related papers  (2021-02-01T12:43:03Z)
• Region Comparison Network for Interpretable Few-shot Image Classification [97.97902360117368]
  Few-shot image classification has been proposed to effectively use only a limited number of labeled examples to train models for new classes. We propose a metric learning based method named Region Comparison Network (RCN), which is able to reveal how few-shot learning works. We also present a new way to generalize the interpretability from the level of tasks to categories.
  arXiv  Detail & Related papers  (2020-09-08T07:29:05Z)
• Zero-Shot Recognition through Image-Guided Semantic Classification [9.291055558504588]
  We present a new embedding-based framework for zero-shot learning (ZSL) Motivated by the binary relevance method for multi-label classification, we propose to inversely learn the mapping between an image and a semantic classifier. IGSC is conceptually simple and can be realized by a slight enhancement of an existing deep architecture for classification.
  arXiv  Detail & Related papers  (2020-07-23T06:22:40Z)
• Learning to Compose Hypercolumns for Visual Correspondence [57.93635236871264]
  We introduce a novel approach to visual correspondence that dynamically composes effective features by leveraging relevant layers conditioned on the images to match. The proposed method, dubbed Dynamic Hyperpixel Flow, learns to compose hypercolumn features on the fly by selecting a small number of relevant layers from a deep convolutional neural network.
  arXiv  Detail & Related papers  (2020-07-21T04:03:22Z)
• Weakly-supervised Object Localization for Few-shot Learning and Fine-grained Few-shot Learning [0.5156484100374058]
  Few-shot learning aims to learn novel visual categories from very few samples. We propose a Self-Attention Based Complementary Module (SAC Module) to fulfill the weakly-supervised object localization. We also produce the activated masks for selecting discriminative deep descriptors for few-shot classification.
  arXiv  Detail & Related papers  (2020-03-02T14:07:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.