Enhancing Vision-Language Few-Shot Adaptation with Negative Learning

• URL: http://arxiv.org/abs/2403.12964v2
• Date: Fri, 08 Nov 2024 14:58:29 GMT
• Title: Enhancing Vision-Language Few-Shot Adaptation with Negative Learning
• Authors: Ce Zhang, Simon Stepputtis, Katia Sycara, Yaqi Xie,
• Abstract summary: We propose a Simple yet effective Negative Learning approach, SimNL, to more efficiently exploit task-specific knowledge. To this issue, we introduce a plug-and-play few-shot instance reweighting technique to mitigate noisy outliers. Our extensive experimental results validate that the proposed SimNL outperforms existing state-of-the-art methods on both few-shot learning and domain generalization tasks.
• Score: 11.545127156146368
• License:
• Abstract: Large-scale pre-trained Vision-Language Models (VLMs) have exhibited impressive zero-shot performance and transferability, allowing them to adapt to downstream tasks in a data-efficient manner. However, when only a few labeled samples are available, adapting VLMs to distinguish subtle differences between similar classes in specific downstream tasks remains challenging. In this work, we propose a Simple yet effective Negative Learning approach, SimNL, to more efficiently exploit the task-specific knowledge from few-shot labeled samples. Unlike previous methods that focus on identifying a set of representative positive features defining "what is a {CLASS}", SimNL discovers a complementary set of negative features that define "what is not a {CLASS}", providing additional insights that supplement the positive features to enhance task-specific recognition capability. Further, we identify that current adaptation approaches are particularly vulnerable to potential noise in the few-shot sample set. To mitigate this issue, we introduce a plug-and-play few-shot instance reweighting technique to suppress noisy outliers and amplify clean samples for more stable adaptation. Our extensive experimental results across 15 datasets validate that the proposed SimNL outperforms existing state-of-the-art methods on both few-shot learning and domain generalization tasks while achieving competitive computational efficiency. Code is available at https://github.com/zhangce01/SimNL.

Related papers

• Active Prompt Learning with Vision-Language Model Priors [9.173468790066956]
  We introduce a class-guided clustering that leverages the pre-trained image and text encoders of vision-language models. We propose a budget-saving selective querying based on adaptive class-wise thresholds.
  arXiv  Detail & Related papers  (2024-11-23T02:34:33Z)
• Active Learning for Vision-Language Models [29.309503214127016]
  We propose a novel active learning (AL) framework that enhances the zero-shot classification performance of vision-language models (VLMs) Our approach first calibrates the predicted entropy of VLMs and then utilizes a combination of self-uncertainty and neighbor-aware uncertainty to calculate a reliable uncertainty measure for active sample selection. Our experiments show that the proposed approach outperforms existing AL approaches on several image classification datasets.
  arXiv  Detail & Related papers  (2024-10-29T16:25:50Z)
• Data Adaptive Traceback for Vision-Language Foundation Models in Image Classification [34.37262622415682]
  We propose a new adaptation framework called Data Adaptive Traceback. Specifically, we utilize a zero-shot-based method to extract the most downstream task-related subset of the pre-training data. We adopt a pseudo-label-based semi-supervised technique to reuse the pre-training images and a vision-language contrastive learning method to address the confirmation bias issue in semi-supervised learning.
  arXiv  Detail & Related papers  (2024-07-11T18:01:58Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning [45.25602203155762]
  Self-Supervised Contrastive Learning has proven effective in deriving high-quality representations from unlabeled data. A major challenge that hinders both unimodal and multimodal contrastive learning is feature suppression. We propose a novel model-agnostic Multistage Contrastive Learning framework.
  arXiv  Detail & Related papers  (2024-02-19T04:13:33Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• Active Learning Principles for In-Context Learning with Large Language Models [65.09970281795769]
  This paper investigates how Active Learning algorithms can serve as effective demonstration selection methods for in-context learning. We show that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.
  arXiv  Detail & Related papers  (2023-05-23T17:16:04Z)
• A Practical Contrastive Learning Framework for Single-Image Super-Resolution [51.422185656787285]
  We investigate contrastive learning-based single image super-resolution from two perspectives. We propose a practical contrastive learning framework for SISR, named PCL-SR. Compared with existing benchmark methods, we re-train them by our proposed PCL-SR framework and achieve superior performance.
  arXiv  Detail & Related papers  (2021-11-27T15:42:12Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• Revisiting Unsupervised Meta-Learning: Amplifying or Compensating for the Characteristics of Few-Shot Tasks [30.893785366366078]
  We develop a practical approach towards few-shot image classification, where a visual recognition system is constructed with limited data. We find that the base class set labels are not necessary, and discriminative embeddings could be meta-learned in an unsupervised manner. Experiments on few-shot learning benchmarks verify our approaches outperform previous methods by a 4-10% performance gap.
  arXiv  Detail & Related papers  (2020-11-30T10:08:35Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.