CRoF: CLIP-based Robust Few-shot Learning on Noisy Labels

• URL: http://arxiv.org/abs/2412.12793v1
• Date: Tue, 17 Dec 2024 10:56:18 GMT
• Title: CRoF: CLIP-based Robust Few-shot Learning on Noisy Labels
• Authors: Shizhuo Deng, Bowen Han, Jiaqi Chen, Hao Wang, Dongyue Chen, Tong Jia,
• Abstract summary: Noisy labels threaten the robustness of few-shot learning due to the inexact features in a new domain.<n>We provide a novel view to mitigate the influence of noisy labels, CLIP-based Robust Few-shot learning (CRoF)<n>CRoF is a general plug-in module for CLIP-based models.
• Score: 12.69583354123737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Noisy labels threaten the robustness of few-shot learning (FSL) due to the inexact features in a new domain. CLIP, a large-scale vision-language model, performs well in FSL on image-text embedding similarities, but it is susceptible to misclassification caused by noisy labels. How to enhance domain generalization of CLIP on noisy data within FSL tasks is a critical challenge. In this paper, we provide a novel view to mitigate the influence of noisy labels, CLIP-based Robust Few-shot learning (CRoF). CRoF is a general plug-in module for CLIP-based models. To avoid misclassification and confused label embedding, we design the few-shot task-oriented prompt generator to give more discriminative descriptions of each category. The proposed prompt achieves larger distances of inter-class textual embedding. Furthermore, rather than fully trusting zero-shot classification by CLIP, we fine-tune CLIP on noisy few-shot data in a new domain with a weighting strategy like label-smooth. The weights for multiple potentially correct labels consider the relationship between CLIP's prior knowledge and original label information to ensure reliability. Our multiple label loss function further supports robust training under this paradigm. Comprehensive experiments show that CRoF, as a plug-in, outperforms fine-tuned and vanilla CLIP models on different noise types and noise ratios.

Related papers

• Making Better Mistakes in CLIP-Based Zero-Shot Classification with Hierarchy-Aware Language Prompts [2.967024581564439]
  We introduce making better mistakes in CLIP-based zero-shot classification. Our approach outperforms the related methods in a holistic comparison across five datasets.
  arXiv  Detail & Related papers  (2025-03-04T03:54:50Z)
• Imbalanced Medical Image Segmentation with Pixel-dependent Noisy Labels [23.049622621090453]
  We propose Collaborative Learning with Curriculum Selection (CLCS) to address pixel-dependent noisy labels with class imbalance. CLCS consists of two modules: Curriculum Noisy Label Sample Selection (CNS) and Noise Balance Loss (NBL)
  arXiv  Detail & Related papers  (2025-01-12T00:59:57Z)
• CLIP-FSAC++: Few-Shot Anomaly Classification with Anomaly Descriptor Based on CLIP [22.850815902535988]
  We propose an effective few-shot anomaly classification framework with one-stage training, dubbed CLIP-FSAC++.<n>In anomaly descriptor, image-to-text cross-attention module is used to obtain image-specific text embeddings.<n> Comprehensive experiment results are provided for evaluating our method in few-normal shot anomaly classification on VisA and MVTEC-AD for 1, 2, 4 and 8-shot settings.
  arXiv  Detail & Related papers  (2024-12-05T02:44:45Z)
• CLIPCleaner: Cleaning Noisy Labels with CLIP [36.434849361479316]
  textitCLIPCleaner is a zero-shot classifier for efficient, offline, clean sample selection. textitCLIPCleaner offers a simple, single-step approach that achieves competitive or superior performance on benchmark datasets.
  arXiv  Detail & Related papers  (2024-08-19T14:05:58Z)
• TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Training [29.431698321195814]
  Contrastive Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in open-vocabulary classification. CLIP shows poor performance on multi-label datasets because the global feature tends to be dominated by the most prominent class. We propose a local-to-global framework to obtain image tags.
  arXiv  Detail & Related papers  (2023-12-20T08:15:40Z)
• Roll With the Punches: Expansion and Shrinkage of Soft Label Selection for Semi-supervised Fine-Grained Learning [42.71454054383897]
  We propose Soft Label Selection with Confidence-Aware Clustering based on Class Transition Tracking (SoC) Our approach demonstrates its superior performance in SS-FGVC.
  arXiv  Detail & Related papers  (2023-12-19T15:22:37Z)
• Combating Label Noise With A General Surrogate Model For Sample Selection [84.61367781175984]
  We propose to leverage the vision-language surrogate model CLIP to filter noisy samples automatically. We validate the effectiveness of our proposed method on both real-world and synthetic noisy datasets.
  arXiv  Detail & Related papers  (2023-10-16T14:43:27Z)
• Channel-Wise Contrastive Learning for Learning with Noisy Labels [60.46434734808148]
  We introduce channel-wise contrastive learning (CWCL) to distinguish authentic label information from noise. Unlike conventional instance-wise contrastive learning (IWCL), CWCL tends to yield more nuanced and resilient features aligned with the authentic labels. Our strategy is twofold: firstly, using CWCL to extract pertinent features to identify cleanly labeled samples, and secondly, progressively fine-tuning using these samples.
  arXiv  Detail & Related papers  (2023-08-14T06:04:50Z)
• Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels [61.97359362447732]
  Learning from noisy labels is an important and long-standing problem in machine learning for real applications. In this paper, we reformulate the label-noise problem from a generative-model perspective. Our model achieves new state-of-the-art (SOTA) results on all the standard real-world benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-31T03:01:36Z)
• PatchMix Augmentation to Identify Causal Features in Few-shot Learning [55.64873998196191]
  Few-shot learning aims to transfer knowledge learned from base with sufficient categories labelled data to novel categories with scarce known information. We propose a novel data augmentation strategy dubbed as PatchMix that can break this spurious dependency. We show that such an augmentation mechanism, different from existing ones, is able to identify the causal features.
  arXiv  Detail & Related papers  (2022-11-29T08:41:29Z)
• Transductive CLIP with Class-Conditional Contrastive Learning [68.51078382124331]
  We propose Transductive CLIP, a novel framework for learning a classification network with noisy labels from scratch. A class-conditional contrastive learning mechanism is proposed to mitigate the reliance on pseudo labels. ensemble labels is adopted as a pseudo label updating strategy to stabilize the training of deep neural networks with noisy labels.
  arXiv  Detail & Related papers  (2022-06-13T14:04:57Z)
• The Role of Global Labels in Few-Shot Classification and How to Infer Them [55.64429518100676]
  Few-shot learning is a central problem in meta-learning, where learners must quickly adapt to new tasks. We propose Meta Label Learning (MeLa), a novel algorithm that infers global labels and obtains robust few-shot models via standard classification.
  arXiv  Detail & Related papers  (2021-08-09T14:07:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.