Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification

• URL: http://arxiv.org/abs/2307.03133v1
• Date: Thu, 6 Jul 2023 16:59:53 GMT
• Title: Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification
• Authors: Yongcan Yu, Lijun Sheng, Ran He, Jian Liang
• Abstract summary: Test-time adaptation (TTA) is a technique aimed at enhancing the generalization performance of models by leveraging unlabeled samples solely during prediction. We present a benchmark that systematically evaluates 13 prominent TTA methods and their variants on five widely used image classification datasets.
• Score: 77.0114672086012
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time adaptation (TTA) is a technique aimed at enhancing the generalization performance of models by leveraging unlabeled samples solely during prediction. Given the need for robustness in neural network systems when faced with distribution shifts, numerous TTA methods have recently been proposed. However, evaluating these methods is often done under different settings, such as varying distribution shifts, backbones, and designing scenarios, leading to a lack of consistent and fair benchmarks to validate their effectiveness. To address this issue, we present a benchmark that systematically evaluates 13 prominent TTA methods and their variants on five widely used image classification datasets: CIFAR-10-C, CIFAR-100-C, ImageNet-C, DomainNet, and Office-Home. These methods encompass a wide range of adaptation scenarios (e.g. online adaptation v.s. offline adaptation, instance adaptation v.s. batch adaptation v.s. domain adaptation). Furthermore, we explore the compatibility of different TTA methods with diverse network backbones. To implement this benchmark, we have developed a unified framework in PyTorch, which allows for consistent evaluation and comparison of the TTA methods across the different datasets and network architectures. By establishing this benchmark, we aim to provide researchers and practitioners with a reliable means of assessing and comparing the effectiveness of TTA methods in improving model robustness and generalization performance. Our code is available at https://github.com/yuyongcan/Benchmark-TTA.

Related papers

• UniTTA: Unified Benchmark and Versatile Framework Towards Realistic Test-Time Adaptation [66.05528698010697]
  Test-Time Adaptation aims to adapt pre-trained models to the target domain during testing. Researchers have identified various challenging scenarios and developed diverse methods to address these challenges. We propose a Unified Test-Time Adaptation benchmark, which is comprehensive and widely applicable.
  arXiv  Detail & Related papers  (2024-07-29T15:04:53Z)
• Distribution Alignment for Fully Test-Time Adaptation with Dynamic Online Data Streams [19.921480334048756]
  Test-Time Adaptation (TTA) enables adaptation and inference in test data streams with domain shifts from the source. We propose a novel Distribution Alignment loss for TTA. We surpass existing methods in non-i.i.d. scenarios and maintain competitive performance under the ideal i.i.d. assumption.
  arXiv  Detail & Related papers  (2024-07-16T19:33:23Z)
• Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairly [79.07074710460012]
  adversarial vulnerability of deep neural networks (DNNs) has drawn great attention. An increasing number of transfer-based methods have been developed to fool black-box DNN models. We establish a transfer-based attack benchmark (TA-Bench) which implements 30+ methods.
  arXiv  Detail & Related papers  (2023-11-02T15:35:58Z)
• AR-TTA: A Simple Method for Real-World Continual Test-Time Adaptation [1.4530711901349282]
  We propose to validate test-time adaptation methods using datasets for autonomous driving, namely CLAD-C and SHIFT. We observe that current test-time adaptation methods struggle to effectively handle varying degrees of domain shift. We enhance the well-established self-training framework by incorporating a small memory buffer to increase model stability.
  arXiv  Detail & Related papers  (2023-09-18T19:34:23Z)
• pSTarC: Pseudo Source Guided Target Clustering for Fully Test-Time Adaptation [15.621092104244003]
  Test Time Adaptation (TTA) is a pivotal concept in machine learning, enabling models to perform well in real-world scenarios. We propose a novel approach called pseudo Source guided Target Clustering (pSTarC) addressing the relatively unexplored area of TTA under real-world domain shifts.
  arXiv  Detail & Related papers  (2023-09-02T07:13:47Z)
• On Pitfalls of Test-Time Adaptation [82.8392232222119]
  Test-Time Adaptation (TTA) has emerged as a promising approach for tackling the robustness challenge under distribution shifts. We present TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art algorithms, a diverse array of distribution shifts, and two evaluation protocols.
  arXiv  Detail & Related papers  (2023-06-06T09:35:29Z)
• Evaluation of Test-Time Adaptation Under Computational Time Constraints [80.40939405129102]
  Test Time Adaptation (TTA) methods leverage unlabeled data at test time to adapt to distribution shifts. Current evaluation protocols overlook the effect of this extra cost, affecting their real-world applicability. We propose a more realistic evaluation protocol for TTA methods, where data is received in an online fashion from a constant-speed data stream.
  arXiv  Detail & Related papers  (2023-04-10T18:01:47Z)
• Test-time Adaptation in the Dynamic World with Compound Domain Knowledge Management [75.86903206636741]
  Test-time adaptation (TTA) allows the model to adapt itself to novel environments and improve its performance during test time. Several works for TTA have shown promising adaptation performances in continuously changing environments. This paper first presents a robust TTA framework with compound domain knowledge management. We then devise novel regularization which modulates the adaptation rates using domain-similarity between the source and the current target domain.
  arXiv  Detail & Related papers  (2022-12-16T09:02:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.