Each Test Image Deserves A Specific Prompt: Continual Test-Time Adaptation for 2D Medical Image Segmentation

• URL: http://arxiv.org/abs/2311.18363v4
• Date: Sat, 25 May 2024 15:03:43 GMT
• Title: Each Test Image Deserves A Specific Prompt: Continual Test-Time Adaptation for 2D Medical Image Segmentation
• Authors: Ziyang Chen, Yongsheng Pan, Yiwen Ye, Mengkang Lu, Yong Xia,
• Abstract summary: Cross-domain distribution shift is a significant obstacle to deploying the pre-trained semantic segmentation model in real-world applications. Test-time adaptation has proven its effectiveness in tackling the cross-domain distribution shift during inference. We propose the Visual Prompt-based Test-Time Adaptation (VPTTA) method to train a specific prompt for each test image to align the statistics in the batch normalization layers.
• Score: 14.71883381837561
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distribution shift widely exists in medical images acquired from different medical centres and poses a significant obstacle to deploying the pre-trained semantic segmentation model in real-world applications. Test-time adaptation has proven its effectiveness in tackling the cross-domain distribution shift during inference. However, most existing methods achieve adaptation by updating the pre-trained models, rendering them susceptible to error accumulation and catastrophic forgetting when encountering a series of distribution shifts (i.e., under the continual test-time adaptation setup). To overcome these challenges caused by updating the models, in this paper, we freeze the pre-trained model and propose the Visual Prompt-based Test-Time Adaptation (VPTTA) method to train a specific prompt for each test image to align the statistics in the batch normalization layers. Specifically, we present the low-frequency prompt, which is lightweight with only a few parameters and can be effectively trained in a single iteration. To enhance prompt initialization, we equip VPTTA with a memory bank to benefit the current prompt from previous ones. Additionally, we design a warm-up mechanism, which mixes source and target statistics to construct warm-up statistics, thereby facilitating the training process. Extensive experiments demonstrate the superiority of our VPTTA over other state-of-the-art methods on two medical image segmentation benchmark tasks. The code and weights of pre-trained source models are available at https://github.com/Chen-Ziyang/VPTTA.

Related papers

• BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping [64.8477128397529]
  We propose a training-required and training-free test-time adaptation framework. We maintain a light-weight key-value memory for feature retrieval from instance-agnostic historical samples and instance-aware boosting samples. We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets.
  arXiv  Detail & Related papers  (2024-10-20T15:58:43Z)
• DOTA: Distributional Test-Time Adaptation of Vision-Language Models [52.98590762456236]
  Training-free test-time dynamic adapter (TDA) is a promising approach to address this issue. We propose a simple yet effective method for DistributiOnal Test-time Adaptation (Dota) Dota continually estimates the distributions of test samples, allowing the model to continually adapt to the deployment environment.
  arXiv  Detail & Related papers  (2024-09-28T15:03:28Z)
• Embedded Prompt Tuning: Towards Enhanced Calibration of Pretrained Models for Medical Images [18.094731760514264]
  We study the effectiveness of fine-tuning methods when adapting foundation models to medical image classification tasks. We propose the Embedded Prompt Tuning (EPT) method by embedding prompt tokens into the expanded channels. EPT outperforms several state-of-the-art finetuning methods by a significant margin on few-shot medical image classification tasks.
  arXiv  Detail & Related papers  (2024-07-01T06:35:53Z)
• Medical Image Segmentation with InTEnt: Integrated Entropy Weighting for Single Image Test-Time Adaptation [6.964589353845092]
  Test-time adaptation (TTA) refers to adapting a trained model to a new domain during testing. Here, we propose to adapt a medical image segmentation model with only a single unlabeled test image. Our method, validated on 24 source/target domain splits across 3 medical image datasets surpasses the leading method by 2.9% Dice coefficient on average.
  arXiv  Detail & Related papers  (2024-02-14T22:26:07Z)
• Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning [73.75282761503581]
  We propose DiffTPT, which leverages pre-trained diffusion models to generate diverse and informative new data. Our experiments on test datasets with distribution shifts and unseen categories demonstrate that DiffTPT improves the zero-shot accuracy by an average of 5.13%.
  arXiv  Detail & Related papers  (2023-08-11T09:36:31Z)
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models [107.05966685291067]
  We propose test-time prompt tuning (TPT) to learn adaptive prompts on the fly with a single test sample. TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
  arXiv  Detail & Related papers  (2022-09-15T17:55:11Z)
• DLTTA: Dynamic Learning Rate for Test-time Adaptation on Cross-domain Medical Images [56.72015587067494]
  We propose a novel dynamic learning rate adjustment method for test-time adaptation, called DLTTA. Our method achieves effective and fast test-time adaptation with consistent performance improvement over current state-of-the-art test-time adaptation methods.
  arXiv  Detail & Related papers  (2022-05-27T02:34:32Z)
• On-the-Fly Test-time Adaptation for Medical Image Segmentation [63.476899335138164]
  Adapting the source model to target data distribution at test-time is an efficient solution for the data-shift problem. We propose a new framework called Adaptive UNet where each convolutional block is equipped with an adaptive batch normalization layer. During test-time, the model takes in just the new test image and generates a domain code to adapt the features of source model according to the test data.
  arXiv  Detail & Related papers  (2022-03-10T18:51:29Z)
• A Field of Experts Prior for Adapting Neural Networks at Test Time [8.244295783641396]
  Performance of convolutional neural networks (CNNs) in image analysis tasks is often marred by acquisition-related distribution shifts between training and test images. It has been proposed to tackle this problem by fine-tuning trained CNNs for each test image. We propose to carry out test-time-adaptation (TTA) by matching the feature distributions of test and training images.
  arXiv  Detail & Related papers  (2022-02-10T11:44:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.