Vision-Language Models are Strong Noisy Label Detectors

• URL: http://arxiv.org/abs/2409.19696v1
• Date: Sun, 29 Sep 2024 12:55:17 GMT
• Title: Vision-Language Models are Strong Noisy Label Detectors
• Authors: Tong Wei, Hao-Tian Li, Chun-Shu Li, Jiang-Xin Shi, Yu-Feng Li, Min-Ling Zhang,
• Abstract summary: This paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification.
• Score: 76.07846780815794
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a significant obstacle during the fine-tuning process. To address this challenge, this paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. The proposed framework establishes a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive features of the class, while the negative prompt serves as a learnable threshold for separating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As a general framework, DeFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean samples. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification.

Related papers

• In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model [13.983810804606264]
  We propose In-Context Prompt Learning (InCPL) for test-time visual recognition tasks. InCPL associates a new test sample with very few labeled examples as context information. We introduce a context-aware unsupervised loss to optimize visual prompts tailored to test samples.
  arXiv  Detail & Related papers  (2024-03-10T08:15:51Z)
• Text as Image: Learning Transferable Adapter for Multi-Label Classification [13.11583340598517]
  We introduce an effective approach to employ large language models for multi-label instruction-following text generation. In this way, a fully automated pipeline for visual label recognition is developed without relying on any manual data.
  arXiv  Detail & Related papers  (2023-12-07T09:22:20Z)
• Combating Label Noise With A General Surrogate Model For Sample Selection [84.61367781175984]
  We propose to leverage the vision-language surrogate model CLIP to filter noisy samples automatically. We validate the effectiveness of our proposed method on both real-world and synthetic noisy datasets.
  arXiv  Detail & Related papers  (2023-10-16T14:43:27Z)
• CAPro: Webly Supervised Learning with Cross-Modality Aligned Prototypes [93.71909293023663]
  Cross-modality Aligned Prototypes (CAPro) is a unified contrastive learning framework to learn visual representations with correct semantics. CAPro achieves new state-of-the-art performance and exhibits robustness to open-set recognition.
  arXiv  Detail & Related papers  (2023-10-15T07:20:22Z)
• Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels? [41.56585313495218]
  A vision-language model can be adapted to a new classification task through few-shot prompt tuning. We study the key reasons contributing to the robustness of the prompt tuning paradigm. We demonstrate that noisy zero-shot predictions from CLIP can be used to tune its own prompt.
  arXiv  Detail & Related papers  (2023-07-22T04:20:30Z)
• Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models [48.77653835765705]
  We introduce a probabilistic resolution to prompt tuning, where the label-specific prompts are generated hierarchically by first sampling a latent vector from an underlying distribution and then employing a lightweight generative model. We evaluate the effectiveness of our approach on four tasks: few-shot image recognition, base-to-new generalization, dataset transfer learning, and domain shifts.
  arXiv  Detail & Related papers  (2023-03-16T06:09:15Z)
• Learning to Detect Noisy Labels Using Model-Based Features [16.681748918518075]
  We propose Selection-Enhanced Noisy label Training (SENT) SENT does not rely on meta learning while having the flexibility of being data-driven. It improves performance over strong baselines under the settings of self-training and label corruption.
  arXiv  Detail & Related papers  (2022-12-28T10:12:13Z)
• Context-based Virtual Adversarial Training for Text Classification with Noisy Labels [1.9508698179748525]
  We propose context-based virtual adversarial training (ConVAT) to prevent a text classifier from overfitting to noisy labels. Unlike the previous works, the proposed method performs the adversarial training at the context level rather than the inputs. We conduct extensive experiments on four text classification datasets with two types of label noises.
  arXiv  Detail & Related papers  (2022-05-29T14:19:49Z)
• Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis [69.48582264712854]
  We propose a robust learning method to perform robust visual sentiment analysis. Our method relies on an external memory to aggregate and filter noisy labels during training. We establish a benchmark for visual sentiment analysis with label noise using publicly available datasets.
  arXiv  Detail & Related papers  (2021-09-15T18:18:28Z)
• Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels [51.66448070984615]
  We show that maximizing training accuracy on sufficiently many noisy samples yields an approximately optimal classifier. For validation, we prove that a noisy validation set is reliable, addressing the critical demand of model selection. We show characterizations of models trained with noisy labels, motivated by our theoretical results, and verify the utility of a noisy validation set.
  arXiv  Detail & Related papers  (2020-12-08T03:37:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.