Align Your Prompts: Test-Time Prompting with Distribution Alignment for
Zero-Shot Generalization
- URL: http://arxiv.org/abs/2311.01459v2
- Date: Thu, 11 Jan 2024 04:32:05 GMT
- Title: Align Your Prompts: Test-Time Prompting with Distribution Alignment for
Zero-Shot Generalization
- Authors: Jameel Hassan, Hanan Gani, Noor Hussein, Muhammad Uzair Khattak,
Muzammal Naseer, Fahad Shahbaz Khan and Salman Khan
- Abstract summary: We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain.
Our method improves zero-shot top- 1 accuracy beyond existing prompt-learning techniques, with a 3.08% improvement over the baseline MaPLe.
- Score: 64.62570402941387
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: The promising zero-shot generalization of vision-language models such as CLIP
has led to their adoption using prompt learning for numerous downstream tasks.
Previous works have shown test-time prompt tuning using entropy minimization to
adapt text prompts for unseen domains. While effective, this overlooks the key
cause for performance degradation to unseen domains -- distribution shift. In
this work, we explicitly handle this problem by aligning the
out-of-distribution (OOD) test sample statistics to those of the source data
using prompt tuning. We use a single test sample to adapt multi-modal prompts
at test time by minimizing the feature distribution shift to bridge the gap in
the test domain. Evaluating against the domain generalization benchmark, our
method improves zero-shot top- 1 accuracy beyond existing prompt-learning
techniques, with a 3.08% improvement over the baseline MaPLe. In cross-dataset
generalization with unseen categories across 10 datasets, our method improves
consistently across all datasets compared to the existing state-of-the-art. Our
source code and models are available at
https://jameelhassan.github.io/promptalign.
Related papers
- PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination [14.50214193838818]
A zero-shot generalization in vision-language (V-L) models such as CLIP has spurred their widespread adoption.
Previous methods have employed test-time prompt tuning to adapt the model to unseen domains, but they overlooked the issue of imbalanced class distributions.
In this study, we employ class-aware prototype alignment weighted by mean class probabilities obtained for a test sample and filtered augmented views.
arXiv Detail & Related papers (2024-04-11T07:26:00Z) - Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models [19.683461002518147]
Test-Time Prototype Shifting (TPS) is a pioneering approach designed to adapt vision-language models to test datasets using unlabeled test inputs.
TPS not only facilitates optimization-free prototype reuse for subsequent predictions but also enables seamless integration with current advancements in prompt engineering.
A notable aspect of our framework is its significantly reduced memory and computational demands when compared to conventional text-prompt tuning methods.
arXiv Detail & Related papers (2024-03-19T17:54:34Z) - Feature Alignment and Uniformity for Test Time Adaptation [8.209137567840811]
Test time adaptation aims to adapt deep neural networks when receiving out of distribution test domain samples.
In this setting, the model can only access online unlabeled test samples and pre-trained models on the training domains.
arXiv Detail & Related papers (2023-03-20T06:44:49Z) - Addressing Distribution Shift at Test Time in Pre-trained Language
Models [3.655021726150369]
State-of-the-art pre-trained language models (PLMs) outperform other models when applied to the majority of language processing tasks.
PLMs have been found to degrade in performance under distribution shift.
We present an approach that improves the performance of PLMs at test-time under distribution shift.
arXiv Detail & Related papers (2022-12-05T16:04:54Z) - TeST: Test-time Self-Training under Distribution Shift [99.68465267994783]
Test-Time Self-Training (TeST) is a technique that takes as input a model trained on some source data and a novel data distribution at test time.
We find that models adapted using TeST significantly improve over baseline test-time adaptation algorithms.
arXiv Detail & Related papers (2022-09-23T07:47:33Z) - Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language
Models [107.05966685291067]
We propose test-time prompt tuning (TPT) to learn adaptive prompts on the fly with a single test sample.
TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average.
In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
arXiv Detail & Related papers (2022-09-15T17:55:11Z) - CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time.
We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
arXiv Detail & Related papers (2022-06-01T03:02:07Z) - Prompt Consistency for Zero-Shot Task Generalization [118.81196556175797]
In this paper, we explore methods to utilize unlabeled data to improve zero-shot performance.
Specifically, we take advantage of the fact that multiple prompts can be used to specify a single task, and propose to regularize prompt consistency.
Our approach outperforms the state-of-the-art zero-shot learner, T0, on 9 out of 11 datasets across 4 NLP tasks by up to 10.6 absolute points in terms of accuracy.
arXiv Detail & Related papers (2022-04-29T19:18:37Z) - Adaptive Risk Minimization: Learning to Adapt to Domain Shift [109.87561509436016]
A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution.
In this work, we consider the problem setting of domain generalization, where the training data are structured into domains and there may be multiple test time shifts.
We introduce the framework of adaptive risk minimization (ARM), in which models are directly optimized for effective adaptation to shift by learning to adapt on the training domains.
arXiv Detail & Related papers (2020-07-06T17:59:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.