MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models

• URL: http://arxiv.org/abs/2512.12268v1
• Date: Sat, 13 Dec 2025 10:23:10 GMT
• Title: MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
• Authors: Yuqing Lei, Yingjun Du, Yawen Huang, Xiantong Zhen, Ling Shao,
• Abstract summary: We propose Meta Test-Time Prompt Tuning (MetaTPT), a meta-learning framework that learns a self-supervised auxiliary task to guide test-time prompt tuning.<n>By coupling augmentation learning with prompt tuning, MetaTPT improves test-time adaptation under domain shifts.
• Score: 62.20230218401528
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) such as CLIP exhibit strong zero-shot generalization but remain sensitive to domain shifts at test time. Test-time prompt tuning (TPT) mitigates this issue by adapting prompts with fixed augmentations, which may falter in more challenging settings. In this work, we propose Meta Test-Time Prompt Tuning (MetaTPT), a meta-learning framework that learns a self-supervised auxiliary task to guide test-time prompt tuning. The auxiliary task dynamically learns parameterized augmentations for each sample, enabling more expressive transformations that capture essential features in target domains. MetaTPT adopts a dual-loop optimization paradigm: an inner loop learns a self-supervised task that generates informative views, while the outer loop performs prompt tuning by enforcing consistency across these views. By coupling augmentation learning with prompt tuning, MetaTPT improves test-time adaptation under domain shifts. Extensive experiments demonstrate that MetaTPT achieves state-of-the-art performance on domain generalization and cross-dataset benchmarks.

Related papers

• DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers [13.964106147449051]
  We leverage metric learning techniques to investigate how the distribution of prompts affects fine-tuning performance.<n>We propose a novel framework, Distribution Aware Visual Prompt Tuning (DA-VPT), to guide the distributions of the prompts.<n>Our method demonstrates that the prompts can serve as an effective bridge to share semantic information between image patches and the class token.
  arXiv  Detail & Related papers  (2025-05-29T17:31:26Z)
• IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting [45.392113146092306]
  We tackle the challenge of optimizing prompt designs for diverse tasks in Multi-Domain Task Incremental Learning.<n>Our Instance-Aware Gated Prompting (IA-GP) strategy enhances adaptation to new tasks while mitigating forgetting.<n>Our Instance-Aware Class-Distribution-Driven Prompting (IA-CDDP) improves the task adaptation process by determining an accurate task-label-related confidence score for each instance.
  arXiv  Detail & Related papers  (2025-03-26T14:59:23Z)
• Historical Test-time Prompt Tuning for Vision Foundation Models [99.96912440427192]
  HisTPT is a Historical Test-time Prompt Tuning technique that memorizes the useful knowledge of the learnt test samples. HisTPT achieves superior prompt tuning performance consistently while handling different visual recognition tasks.
  arXiv  Detail & Related papers  (2024-10-27T06:03:15Z)
• Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models [4.655740975414312]
  This paper introduces Test-Time Low-rank adaptation (TTL) as an alternative to prompt tuning for zero-shot generalizations of large-scale vision-language models (VLMs) TTL offers a test-time-efficient adaptation approach that updates the attention weights of the transformer by maximizing prediction confidence.
  arXiv  Detail & Related papers  (2024-07-22T17:59:19Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Dynamic Prompting: A Unified Framework for Prompt Tuning [33.175097465669374]
  We present a unified dynamic prompt (DP) tuning strategy that dynamically determines different factors of prompts based on specific tasks and instances. Experimental results underscore the significant performance improvement achieved by dynamic prompt tuning across a wide range of tasks. We establish the universal applicability of our approach under full-data, few-shot, and multitask scenarios.
  arXiv  Detail & Related papers  (2023-03-06T06:04:46Z)
• How Does In-Context Learning Help Prompt Tuning? [55.78535874154915]
  Fine-tuning large language models is becoming ever more impractical due to their rapidly-growing scale. This motivates the use of parameter-efficient adaptation methods such as prompt tuning (PT), which adds a small number of tunable embeddings to an otherwise frozen model. Recently, Singhal et al. (2022) propose instruction prompt tuning'' (IPT), which combines PT with ICL by concatenating a natural language demonstration with learned prompt embeddings.
  arXiv  Detail & Related papers  (2023-02-22T17:45:12Z)
• Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning? [37.522581151997734]
  Prompt tuning (PT) which only tunes the embeddings of an additional sequence of tokens per task, has shown remarkable performance in few-shot learning. We study meta prompt tuning (MPT) to explore how meta-learning can help improve (if it can) cross-task generalization.
  arXiv  Detail & Related papers  (2023-02-16T08:37:22Z)
• Unified Vision and Language Prompt Learning [86.1530128487077]
  We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities.
  arXiv  Detail & Related papers  (2022-10-13T17:50:24Z)
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models [107.05966685291067]
  We propose test-time prompt tuning (TPT) to learn adaptive prompts on the fly with a single test sample. TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
  arXiv  Detail & Related papers  (2022-09-15T17:55:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.