Dynamic Prompt Allocation and Tuning for Continual Test-Time Adaptation

• URL: http://arxiv.org/abs/2412.09308v1
• Date: Thu, 12 Dec 2024 14:24:04 GMT
• Title: Dynamic Prompt Allocation and Tuning for Continual Test-Time Adaptation
• Authors: Chaoran Cui, Yongrui Zhen, Shuai Gong, Chunyun Zhang, Hui Liu, Yilong Yin,
• Abstract summary: Continual test-time adaptation (CTTA) has recently emerged to adapt to continuously evolving target distributions. Existing methods typically incorporate explicit regularization terms to constrain the variation of model parameters. We introduce learnable domain-specific prompts that guide the model to adapt to corresponding target domains.
• Score: 29.931721498877483
• License:
• Abstract: Continual test-time adaptation (CTTA) has recently emerged to adapt a pre-trained source model to continuously evolving target distributions, which accommodates the dynamic nature of real-world environments. To mitigate the risk of catastrophic forgetting in CTTA, existing methods typically incorporate explicit regularization terms to constrain the variation of model parameters. However, they cannot fundamentally resolve catastrophic forgetting because they rely on a single shared model to adapt across all target domains, which inevitably leads to severe inter-domain interference. In this paper, we introduce learnable domain-specific prompts that guide the model to adapt to corresponding target domains, thereby partially disentangling the parameter space of different domains. In the absence of domain identity for target samples, we propose a novel dynamic Prompt AllocatIon aNd Tuning (PAINT) method, which utilizes a query mechanism to dynamically determine whether the current samples come from a known domain or an unexplored one. For known domains, the corresponding domain-specific prompt is directly selected, while for previously unseen domains, a new prompt is allocated. Prompt tuning is subsequently performed using mutual information maximization along with structural regularization. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our PAINT method for CTTA. We have released our code at https://github.com/Cadezzyr/PAINT.

Related papers

• DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation [11.151967974753925]
  Continual Test-Time Adaptation (CTTA) seeks to adapt source pre-trained models to continually changing, unseen target domains. DPCore is a method designed for robust performance across diverse domain change patterns.
  arXiv  Detail & Related papers  (2024-06-15T20:47:38Z)
• StyDeSty: Min-Max Stylization and Destylization for Single Domain Generalization [85.18995948334592]
  Single domain generalization (single DG) aims at learning a robust model generalizable to unseen domains from only one training domain. State-of-the-art approaches have mostly relied on data augmentations, such as adversarial perturbation and style enhancement, to synthesize new data. We propose emphStyDeSty, which explicitly accounts for the alignment of the source and pseudo domains in the process of data augmentation.
  arXiv  Detail & Related papers  (2024-06-01T02:41:34Z)
• Memory-Efficient Prompt Tuning for Incremental Histopathology Classification [69.46798702300042]
  We present a memory-efficient prompt tuning framework to cultivate model generalization potential in economical memory cost. We have extensively evaluated our framework with two histopathology tasks, i.e., breast cancer metastasis classification and epithelium-stroma tissue classification.
  arXiv  Detail & Related papers  (2024-01-22T03:24:45Z)
• Revisiting the Domain Shift and Sample Uncertainty in Multi-source Active Domain Transfer [69.82229895838577]
  Active Domain Adaptation (ADA) aims to maximally boost model adaptation in a new target domain by actively selecting a limited number of target data to annotate. This setting neglects the more practical scenario where training data are collected from multiple sources. This motivates us to target a new and challenging setting of knowledge transfer that extends ADA from a single source domain to multiple source domains.
  arXiv  Detail & Related papers  (2023-11-21T13:12:21Z)
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation [48.039156140237615]
  A Continual Test-Time Adaptation task is proposed to adapt the pre-trained model to continually changing target domains. We design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain-specific and domain-shared knowledge. Our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks.
  arXiv  Detail & Related papers  (2023-06-07T11:18:53Z)
• Self-Paced Learning for Open-Set Domain Adaptation [50.620824701934]
  Traditional domain adaptation methods presume that the classes in the source and target domains are identical. Open-set domain adaptation (OSDA) addresses this limitation by allowing previously unseen classes in the target domain. We propose a novel framework based on self-paced learning to distinguish common and unknown class samples.
  arXiv  Detail & Related papers  (2023-03-10T14:11:09Z)
• Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation [14.473807945791132]
  Continual Test-Time Adaptation (CTTA) aims to adapt the source model to continually changing unlabeled target domains without access to the source data. Motivated by the prompt learning in NLP, in this paper we propose to learn an image-level visual domain prompt for target domains while having the source model parameters frozen.
  arXiv  Detail & Related papers  (2022-12-08T08:56:02Z)
• Towards Online Domain Adaptive Object Detection [79.89082006155135]
  Existing object detection models assume both the training and test data are sampled from the same source domain. We propose a novel unified adaptation framework that adapts and improves generalization on the target domain in online settings.
  arXiv  Detail & Related papers  (2022-04-11T17:47:22Z)
• Bi-Directional Generation for Unsupervised Domain Adaptation [61.73001005378002]
  Unsupervised domain adaptation facilitates the unlabeled target domain relying on well-established source domain information. Conventional methods forcefully reducing the domain discrepancy in the latent space will result in the destruction of intrinsic data structure. We propose a Bi-Directional Generation domain adaptation model with consistent classifiers interpolating two intermediate domains to bridge source and target domains.
  arXiv  Detail & Related papers  (2020-02-12T09:45:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.