The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learning

• URL: http://arxiv.org/abs/2404.09778v2
• Date: Fri, 19 Apr 2024 02:19:19 GMT
• Title: The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learning
• Authors: Yaohui Li, Qifeng Zhou, Haoxing Chen, Jianbing Zhang, Xinyu Dai, Hao Zhou,
• Abstract summary: Contrastive Language-Image Pre-training (CLIP) has shown powerful zero-shot learning performance. Few-shot learning aims to further enhance the transfer capability of CLIP by giving few images in each class, aka 'few shots'
• Score: 29.87420015681205
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive Language-Image Pre-training (CLIP) has shown powerful zero-shot learning performance. Few-shot learning aims to further enhance the transfer capability of CLIP by giving few images in each class, aka 'few shots'. Most existing methods either implicitly learn from the few shots by incorporating learnable prompts or adapters, or explicitly embed them in a cache model for inference. However, the narrow distribution of few shots often contains incomplete class information, leading to biased visual knowledge with high risk of misclassification. To tackle this problem, recent methods propose to supplement visual knowledge by generative models or extra databases, which can be costly and time-consuming. In this paper, we propose an Iterative Visual Knowledge CompLetion (KCL) method to complement visual knowledge by properly taking advantages of unlabeled samples without access to any auxiliary or synthetic data. Specifically, KCL first measures the similarities between unlabeled samples and each category. Then, the samples with top confidence to each category is selected and collected by a designed confidence criterion. Finally, the collected samples are treated as labeled ones and added to few shots to jointly re-estimate the remaining unlabeled ones. The above procedures will be repeated for a certain number of iterations with more and more samples being collected until convergence, ensuring a progressive and robust knowledge completion process. Extensive experiments on 11 benchmark datasets demonstrate the effectiveness and efficiency of KCL as a plug-and-play module under both few-shot and zero-shot learning settings. Code is available at https://github.com/Mark-Sky/KCL.

Related papers

• Transductive Zero-Shot and Few-Shot CLIP [24.592841797020203]
  This paper addresses the transductive zero-shot and few-shot CLIP classification challenge. Inference is performed jointly across a mini-batch of unlabeled query samples, rather than treating each instance independently. Our approach yields near 20% improvement in ImageNet accuracy over CLIP's zero-shot performance.
  arXiv  Detail & Related papers  (2024-04-08T12:44:31Z)
• Enhancing Visual Continual Learning with Language-Guided Supervision [76.38481740848434]
  Continual learning aims to empower models to learn new tasks without forgetting previously acquired knowledge. We argue that the scarce semantic information conveyed by the one-hot labels hampers the effective knowledge transfer across tasks. Specifically, we use PLMs to generate semantic targets for each class, which are frozen and serve as supervision signals.
  arXiv  Detail & Related papers  (2024-03-24T12:41:58Z)
• Decoupled Contrastive Learning for Long-Tailed Recognition [58.255966442426484]
  Supervised Contrastive Loss (SCL) is popular in visual representation learning. In the scenario of long-tailed recognition, where the number of samples in each class is imbalanced, treating two types of positive samples equally leads to the biased optimization for intra-category distance. We propose a patch-based self distillation to transfer knowledge from head to tail classes to relieve the under-representation of tail classes.
  arXiv  Detail & Related papers  (2024-03-10T09:46:28Z)
• Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learning [56.29097276129473]
  We propose a simple yet effective framework, named Learning Prompt with Distribution-based Feature Replay (LP-DiF) To prevent the learnable prompt from forgetting old knowledge in the new session, we propose a pseudo-feature replay approach. When progressing to a new session, pseudo-features are sampled from old-class distributions combined with training images of the current session to optimize the prompt.
  arXiv  Detail & Related papers  (2024-01-03T07:59:17Z)
• Virtual Category Learning: A Semi-Supervised Learning Method for Dense Prediction with Extremely Limited Labels [63.16824565919966]
  This paper proposes to use confusing samples proactively without label correction. A Virtual Category (VC) is assigned to each confusing sample in such a way that it can safely contribute to the model optimisation. Our intriguing findings highlight the usage of VC learning in dense vision tasks.
  arXiv  Detail & Related papers  (2023-12-02T16:23:52Z)
• Few-shot Class-Incremental Semantic Segmentation via Pseudo-Labeling and Knowledge Distillation [3.4436201325139737]
  We address the problem of learning new classes for semantic segmentation models from few examples. For learning from limited data, we propose a pseudo-labeling strategy to augment the few-shot training annotations. We integrate the above steps into a single convolutional neural network with a unified learning objective.
  arXiv  Detail & Related papers  (2023-08-05T05:05:37Z)
• Text-to-Image Diffusion Models are Zero-Shot Classifiers [8.26990105697146]
  We investigate text-to-image diffusion models by proposing a method for evaluating them as zero-shot classifiers. We apply our method to Stable Diffusion and Imagen, using it to probe fine-grained aspects of the models' knowledge. They perform competitively with CLIP on a wide range of zero-shot image classification datasets.
  arXiv  Detail & Related papers  (2023-03-27T14:15:17Z)
• PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery [39.03732147384566]
  Generalized Novel Category Discovery (GNCD) setting aims to categorize unlabeled training data coming from known and novel classes. We propose Contrastive Affinity Learning method with auxiliary visual Prompts, dubbed PromptCAL, to address this challenging problem. Our approach discovers reliable pairwise sample affinities to learn better semantic clustering of both known and novel classes for the class token and visual prompts.
  arXiv  Detail & Related papers  (2022-12-11T20:06:14Z)
• Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification [57.5041270212206]
  We present BAtch Knowledge Ensembling (BAKE) to produce refined soft targets for anchor images. BAKE achieves online knowledge ensembling across multiple samples with only a single network. It requires minimal computational and memory overhead compared to existing knowledge ensembling methods.
  arXiv  Detail & Related papers  (2021-04-27T16:11:45Z)
• Few-Shot Learning with Intra-Class Knowledge Transfer [100.87659529592223]
  We consider the few-shot classification task with an unbalanced dataset. Recent works have proposed to solve this task by augmenting the training data of the few-shot classes using generative models. We propose to leverage the intra-class knowledge from the neighbor many-shot classes with the intuition that neighbor classes share similar statistical information.
  arXiv  Detail & Related papers  (2020-08-22T18:15:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.