Parameter-Selective Continual Test-Time Adaptation

• URL: http://arxiv.org/abs/2407.02253v1
• Date: Tue, 2 Jul 2024 13:18:15 GMT
• Title: Parameter-Selective Continual Test-Time Adaptation
• Authors: Jiaxu Tian, Fan Lyu,
• Abstract summary: Continual Test-Time Adaptation (CTTA) aims to adapt a pretrained model to ever-changing environments during the test time under continuous domain shifts. PSMT method is capable of effectively updating the critical parameters within the MT network under domain shifts.
• Score: 3.480626767752489
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual Test-Time Adaptation (CTTA) aims to adapt a pretrained model to ever-changing environments during the test time under continuous domain shifts. Most existing CTTA approaches are based on the Mean Teacher (MT) structure, which contains a student and a teacher model, where the student is updated using the pseudo-labels from the teacher model, and the teacher is then updated by exponential moving average strategy. However, these methods update the MT model indiscriminately on all parameters of the model. That is, some critical parameters involving sharing knowledge across different domains may be erased, intensifying error accumulation and catastrophic forgetting. In this paper, we introduce Parameter-Selective Mean Teacher (PSMT) method, which is capable of effectively updating the critical parameters within the MT network under domain shifts. First, we introduce a selective distillation mechanism in the student model, which utilizes past knowledge to regularize novel knowledge, thereby mitigating the impact of error accumulation. Second, to avoid catastrophic forgetting, in the teacher model, we create a mask through Fisher information to selectively update parameters via exponential moving average, with preservation measures applied to crucial parameters. Extensive experimental results verify that PSMT outperforms state-of-the-art methods across multiple benchmark datasets. Our code is available at \url{https://github.com/JiaxuTian/PSMT}.

Related papers

• Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data [39.40116554523575]
  We present Drift-Resilient TabPFN, a fresh approach based on In-Context Learning with a Prior-Data Fitted Network. It learns to approximate Bayesian inference on synthetic datasets drawn from a prior. It improves accuracy from 0.688 to 0.744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration.
  arXiv  Detail & Related papers  (2024-11-15T23:49:23Z)
• Sparse Orthogonal Parameters Tuning for Continual Learning [34.462967722928724]
  Continual learning methods based on pre-trained models (PTM) have recently gained attention which adapt to successive downstream tasks without catastrophic forgetting. We propose a novel yet effective method called SoTU (Sparse Orthogonal Parameters TUning)
  arXiv  Detail & Related papers  (2024-11-05T05:19:09Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Adaptive Rentention & Correction for Continual Learning [114.5656325514408]
  A common problem in continual learning is the classification layer's bias towards the most recent task. We name our approach Adaptive Retention & Correction (ARC) ARC achieves an average performance increase of 2.7% and 2.6% on the CIFAR-100 and Imagenet-R datasets.
  arXiv  Detail & Related papers  (2024-05-23T08:43:09Z)
• Variational Continual Test-Time Adaptation [25.262385466354253]
  The prior drift is crucial in Continual Test-Time Adaptation (CTTA) methods that only use unlabeled test data. We introduce VCoTTA, a variational Bayesian approach to measure uncertainties in CTTA. Experimental results on three datasets demonstrate the method's effectiveness in mitigating prior drift.
  arXiv  Detail & Related papers  (2024-02-13T02:41:56Z)
• Periodically Exchange Teacher-Student for Source-Free Object Detection [7.222926042027062]
  Source-free object detection (SFOD) aims to adapt the source detector to unlabeled target domain data in the absence of source domain data. Most SFOD methods follow the same self-training paradigm using mean-teacher (MT) framework where the student model is guided by only one single teacher model. We propose the Periodically Exchange Teacher-Student (PETS) method, a simple yet novel approach that introduces a multiple-teacher framework consisting of a static teacher, a dynamic teacher, and a student model.
  arXiv  Detail & Related papers  (2023-11-23T11:30:54Z)
• A Probabilistic Framework for Lifelong Test-Time Adaptation [34.07074915005366]
  Test-time adaptation (TTA) is the problem of updating a pre-trained source model at inference time given test input(s) from a different target domain. We present PETAL (Probabilistic lifElong Test-time Adaptation with seLf-training prior), which solves lifelong TTA using a probabilistic approach. Our method achieves better results than the current state-of-the-art for online lifelong test-time adaptation across various benchmarks.
  arXiv  Detail & Related papers  (2022-12-19T18:42:19Z)
• TeST: Test-time Self-Training under Distribution Shift [99.68465267994783]
  Test-Time Self-Training (TeST) is a technique that takes as input a model trained on some source data and a novel data distribution at test time. We find that models adapted using TeST significantly improve over baseline test-time adaptation algorithms.
  arXiv  Detail & Related papers  (2022-09-23T07:47:33Z)
• Adapting the Mean Teacher for keypoint-based lung registration under geometric domain shifts [75.51482952586773]
  deep neural networks generally require plenty of labeled training data and are vulnerable to domain shifts between training and test data. We present a novel approach to geometric domain adaptation for image registration, adapting a model from a labeled source to an unlabeled target domain. Our method consistently improves on the baseline model by 50%/47% while even matching the accuracy of models trained on target data.
  arXiv  Detail & Related papers  (2022-07-01T12:16:42Z)
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding [60.226644697970116]
  Domain classification is the fundamental task in natural language understanding (NLU) Most existing continual learning approaches suffer from low accuracy and performance fluctuation. We propose a hyper parameter-free continual learning model for text data that can stably produce high performance under various environments.
  arXiv  Detail & Related papers  (2022-01-05T02:46:16Z)
• Rethinking the Hyperparameters for Fine-tuning [78.15505286781293]
  Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyper parameters. This paper re-examines several common practices of setting hyper parameters for fine-tuning.
  arXiv  Detail & Related papers  (2020-02-19T18:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.