Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting

• URL: http://arxiv.org/abs/2403.11491v1
• Date: Mon, 18 Mar 2024 05:49:45 GMT
• Title: Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting
• Authors: Mingkui Tan, Guohao Chen, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Peilin Zhao, Shuaicheng Niu,
• Abstract summary: Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample. Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications. We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
• Score: 55.17761802332469
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample. Although recent TTA has shown promising performance, we still face two key challenges: 1) prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications; 2) while existing TTA can significantly improve the test performance on out-of-distribution data, they often suffer from severe performance degradation on in-distribution data after TTA (known as forgetting). To this end, we have proposed an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples for test-time entropy minimization. To alleviate forgetting, EATA introduces a Fisher regularizer estimated from test samples to constrain important model parameters from drastic changes. However, in EATA, the adopted entropy loss consistently assigns higher confidence to predictions even for samples that are underlying uncertain, leading to overconfident predictions. To tackle this, we further propose EATA with Calibration (EATA-C) to separately exploit the reducible model uncertainty and the inherent data uncertainty for calibrated TTA. Specifically, we measure the model uncertainty by the divergence between predictions from the full network and its sub-networks, on which we propose a divergence loss to encourage consistent predictions instead of overconfident ones. To further recalibrate prediction confidence, we utilize the disagreement among predicted labels as an indicator of the data uncertainty, and then devise a min-max entropy regularizer to selectively increase and decrease prediction confidence for different samples. Experiments on image classification and semantic segmentation verify the effectiveness of our methods.

Related papers

• DOTA: Distributional Test-Time Adaptation of Vision-Language Models [52.98590762456236]
  Training-free test-time dynamic adapter (TDA) is a promising approach to address this issue. We propose a simple yet effective method for DistributiOnal Test-time Adaptation (Dota) Dota continually estimates the distributions of test samples, allowing the model to continually adapt to the deployment environment.
  arXiv  Detail & Related papers  (2024-09-28T15:03:28Z)
• AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation [7.079932622432037]
  Test-time adaptation (TTA) has emerged as a viable solution to adapt pre-trained models to domain shifts using unlabeled test data. We propose AETTA, a label-free accuracy estimation algorithm for TTA. We show that AETTA shows an average of 19.8%p more accurate estimation compared with the baselines.
  arXiv  Detail & Related papers  (2024-04-01T04:21:49Z)
• Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors [36.54076844195179]
  Test-time adaptation (TTA) fine-tunes pre-trained deep neural networks for unseen test data. We introduce a novel TTA method named Destroy Your Object (DeYO)
  arXiv  Detail & Related papers  (2024-03-12T07:01:57Z)
• Towards Open-Set Test-Time Adaptation Utilizing the Wisdom of Crowds in Entropy Minimization [47.61333493671805]
  Test-time adaptation (TTA) methods rely on the model's predictions to adapt the source pretrained model to the unlabeled target domain. We propose a simple yet effective sample selection method inspired by the following crucial empirical finding.
  arXiv  Detail & Related papers  (2023-08-14T01:24:18Z)
• Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning [73.75282761503581]
  We propose DiffTPT, which leverages pre-trained diffusion models to generate diverse and informative new data. Our experiments on test datasets with distribution shifts and unseen categories demonstrate that DiffTPT improves the zero-shot accuracy by an average of 5.13%.
  arXiv  Detail & Related papers  (2023-08-11T09:36:31Z)
• DELTA: degradation-free fully test-time adaptation [59.74287982885375]
  We find that two unfavorable defects are concealed in the prevalent adaptation methodologies like test-time batch normalization (BN) and self-learning. First, we reveal that the normalization statistics in test-time BN are completely affected by the currently received test samples, resulting in inaccurate estimates. Second, we show that during test-time adaptation, the parameter update is biased towards some dominant classes.
  arXiv  Detail & Related papers  (2023-01-30T15:54:00Z)
• Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
  Test-time adaptation seeks to tackle potential distribution shifts between training and testing data. We propose an active sample selection criterion to identify reliable and non-redundant samples. We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
  arXiv  Detail & Related papers  (2022-04-06T06:39:40Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.