Test-Time Adaptation with Perturbation Consistency Learning
- URL: http://arxiv.org/abs/2304.12764v1
- Date: Tue, 25 Apr 2023 12:29:22 GMT
- Title: Test-Time Adaptation with Perturbation Consistency Learning
- Authors: Yi Su, Yixin Ji, Juntao Li, Hai Ye, Min Zhang
- Abstract summary: We propose a simple test-time adaptation method to promote the model to make stable predictions for samples with distribution shifts.
Our method can achieve higher or comparable performance with less inference time over strong PLM backbones.
- Score: 32.58879780726279
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Currently, pre-trained language models (PLMs) do not cope well with the
distribution shift problem, resulting in models trained on the training set
failing in real test scenarios. To address this problem, the test-time
adaptation (TTA) shows great potential, which updates model parameters to suit
the test data at the testing time. Existing TTA methods rely on well-designed
auxiliary tasks or self-training strategies based on pseudo-label. However,
these methods do not achieve good trade-offs regarding performance gains and
computational costs. To obtain some insights into such a dilemma, we take two
representative TTA methods, i.e., Tent and OIL, for exploration and find that
stable prediction is the key to achieving a good balance. Accordingly, in this
paper, we propose perturbation consistency learning (PCL), a simple test-time
adaptation method to promote the model to make stable predictions for samples
with distribution shifts. Extensive experiments on adversarial robustness and
cross-lingual transferring demonstrate that our method can achieve higher or
comparable performance with less inference time over strong PLM backbones and
previous state-of-the-art TTA methods.
Related papers
- DOTA: Distributional Test-Time Adaptation of Vision-Language Models [52.98590762456236]
Training-free test-time dynamic adapter (TDA) is a promising approach to address this issue.
We propose a simple yet effective method for DistributiOnal Test-time Adaptation (Dota)
Dota continually estimates the distributions of test samples, allowing the model to continually adapt to the deployment environment.
arXiv Detail & Related papers (2024-09-28T15:03:28Z) - Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [55.17761802332469]
Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample.
Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications.
We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
arXiv Detail & Related papers (2024-03-18T05:49:45Z) - A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts [143.14128737978342]
Test-time adaptation, an emerging paradigm, has the potential to adapt a pre-trained model to unlabeled data during testing, before making predictions.
Recent progress in this paradigm highlights the significant benefits of utilizing unlabeled data for training self-adapted models prior to inference.
arXiv Detail & Related papers (2023-03-27T16:32:21Z) - Learning to Adapt to Online Streams with Distribution Shifts [22.155844301575883]
Test-time adaptation (TTA) is a technique used to reduce distribution gaps between the training and testing sets by leveraging unlabeled test data during inference.
In this work, we expand TTA to a more practical scenario, where the test data comes in the form of online streams that experience distribution shifts over time.
We propose a meta-learning approach that teaches the network to adapt to distribution-shifting online streams during meta-training. As a result, the trained model can perform continual adaptation to distribution shifts in testing, regardless of the batch size restriction.
arXiv Detail & Related papers (2023-03-02T23:36:10Z) - DELTA: degradation-free fully test-time adaptation [59.74287982885375]
We find that two unfavorable defects are concealed in the prevalent adaptation methodologies like test-time batch normalization (BN) and self-learning.
First, we reveal that the normalization statistics in test-time BN are completely affected by the currently received test samples, resulting in inaccurate estimates.
Second, we show that during test-time adaptation, the parameter update is biased towards some dominant classes.
arXiv Detail & Related papers (2023-01-30T15:54:00Z) - TeST: Test-time Self-Training under Distribution Shift [99.68465267994783]
Test-Time Self-Training (TeST) is a technique that takes as input a model trained on some source data and a novel data distribution at test time.
We find that models adapted using TeST significantly improve over baseline test-time adaptation algorithms.
arXiv Detail & Related papers (2022-09-23T07:47:33Z) - Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
Test-time adaptation seeks to tackle potential distribution shifts between training and testing data.
We propose an active sample selection criterion to identify reliable and non-redundant samples.
We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
arXiv Detail & Related papers (2022-04-06T06:39:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.