Monitoring Risks in Test-Time Adaptation

• URL: http://arxiv.org/abs/2507.08721v1
• Date: Fri, 11 Jul 2025 16:21:33 GMT
• Title: Monitoring Risks in Test-Time Adaptation
• Authors: Mona Schirmer, Metod Jazbec, Christian A. Naesseth, Eric Nalisnick,
• Abstract summary: Test-time adaptation (TTA) methods address this issue by continuously adapting a deployed model using only unlabeled test data.<n>We propose pairing TTA with risk monitoring frameworks that track predictive performance and raise alerts when predefined performance criteria are violated.<n>Our extensions unlock the application of rigorous statistical risk monitoring to TTA.
• Score: 3.4971919212033877
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Encountering shifted data at test time is a ubiquitous challenge when deploying predictive models. Test-time adaptation (TTA) methods address this issue by continuously adapting a deployed model using only unlabeled test data. While TTA can extend the model's lifespan, it is only a temporary solution. Eventually the model might degrade to the point that it must be taken offline and retrained. To detect such points of ultimate failure, we propose pairing TTA with risk monitoring frameworks that track predictive performance and raise alerts when predefined performance criteria are violated. Specifically, we extend existing monitoring tools based on sequential testing with confidence sequences to accommodate scenarios in which the model is updated at test time and no test labels are available to estimate the performance metrics of interest. Our extensions unlock the application of rigorous statistical risk monitoring to TTA, and we demonstrate the effectiveness of our proposed TTA monitoring framework across a representative set of datasets, distribution shift types, and TTA methods.

Related papers

• R.I.P.: A Simple Black-box Attack on Continual Test-time Adaptation [12.024233973321756]
  Test-time adaptation (TTA) has emerged as a promising solution to tackle the continual domain shift in machine learning.<n>We successfully identify a risk in the sampling process of testing data that could easily degrade the performance of a continual TTA model.
  arXiv  Detail & Related papers  (2024-12-02T05:55:13Z)
• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [55.17761802332469]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample. Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications. We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)
• Persistent Test-time Adaptation in Recurring Testing Scenarios [12.024233973321756]
  Current test-time adaptation (TTA) approaches aim to adapt a machine learning model to environments that change continuously. Yet, it is unclear whether TTA methods can maintain their adaptability over prolonged periods. We propose persistent TTA (PeTTA) which senses when the model is diverging towards collapse and adjusts the adaptation strategy.
  arXiv  Detail & Related papers  (2023-11-30T02:24:44Z)
• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts [117.72709110877939]
  Test-time adaptation (TTA) has the potential to adapt a pre-trained model to unlabeled data during testing, before making predictions.<n>We categorize TTA into several distinct groups based on the form of test data, namely, test-time domain adaptation, test-time batch adaptation, and online test-time adaptation.
  arXiv  Detail & Related papers  (2023-03-27T16:32:21Z)
• Towards Stable Test-Time Adaptation in Dynamic Wild World [60.98073673220025]
  Test-time adaptation (TTA) has shown to be effective at tackling distribution shifts between training and testing data by adapting a given model on test samples. Online model updating of TTA may be unstable and this is often a key obstacle preventing existing TTA methods from being deployed in the real world.
  arXiv  Detail & Related papers  (2023-02-24T02:03:41Z)
• Uncovering Adversarial Risks of Test-Time Adaptation [41.19226800089764]
  Test-time adaptation (TTA) has been proposed as a promising solution for addressing distribution shifts. We uncover a novel security vulnerability of TTA based on the insight that predictions on benign samples can be impacted by malicious samples in the same batch. We propose Distribution Invading Attack (DIA), which injects a small fraction of malicious data into the test batch.
  arXiv  Detail & Related papers  (2023-01-29T22:58:05Z)
• TeST: Test-time Self-Training under Distribution Shift [99.68465267994783]
  Test-Time Self-Training (TeST) is a technique that takes as input a model trained on some source data and a novel data distribution at test time. We find that models adapted using TeST significantly improve over baseline test-time adaptation algorithms.
  arXiv  Detail & Related papers  (2022-09-23T07:47:33Z)
• Robust Continual Test-time Adaptation: Instance-aware BN and Prediction-balanced Memory [58.72445309519892]
  We present a new test-time adaptation scheme that is robust against non-i.i.d. test data streams. Our novelty is mainly two-fold: (a) Instance-Aware Batch Normalization (IABN) that corrects normalization for out-of-distribution samples, and (b) Prediction-balanced Reservoir Sampling (PBRS) that simulates i.i.d. data stream from non-i.i.d. stream in a class-balanced manner.
  arXiv  Detail & Related papers  (2022-08-10T03:05:46Z)
• Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
  Test-time adaptation seeks to tackle potential distribution shifts between training and testing data. We propose an active sample selection criterion to identify reliable and non-redundant samples. We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
  arXiv  Detail & Related papers  (2022-04-06T06:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.