Noisy Test-Time Adaptation in Vision-Language Models

• URL: http://arxiv.org/abs/2502.14604v1
• Date: Thu, 20 Feb 2025 14:37:53 GMT
• Title: Noisy Test-Time Adaptation in Vision-Language Models
• Authors: Chentao Cao, Zhun Zhong, Zhanke Zhou, Tongliang Liu, Yang Liu, Kun Zhang, Bo Han,
• Abstract summary: Test-time adaptation (TTA) aims to address distribution shifts between source and target data by relying solely on target data during testing. This paper introduces Zero-Shot Noisy TTA (ZS-NTTA), focusing on adapting the model to target data with noisy samples during test-time in a zero-shot manner. We introduce the Adaptive Noise Detector (AdaND), which utilizes the frozen model's outputs as pseudo-labels to train a noise detector.
• Score: 73.14136220844156
• License:
• Abstract: Test-time adaptation (TTA) aims to address distribution shifts between source and target data by relying solely on target data during testing. In open-world scenarios, models often encounter noisy samples, i.e., samples outside the in-distribution (ID) label space. Leveraging the zero-shot capability of pre-trained vision-language models (VLMs), this paper introduces Zero-Shot Noisy TTA (ZS-NTTA), focusing on adapting the model to target data with noisy samples during test-time in a zero-shot manner. We find existing TTA methods underperform under ZS-NTTA, often lagging behind even the frozen model. We conduct comprehensive experiments to analyze this phenomenon, revealing that the negative impact of unfiltered noisy data outweighs the benefits of clean data during model updating. Also, adapting a classifier for ID classification and noise detection hampers both sub-tasks. Built on this, we propose a framework that decouples the classifier and detector, focusing on developing an individual detector while keeping the classifier frozen. Technically, we introduce the Adaptive Noise Detector (AdaND), which utilizes the frozen model's outputs as pseudo-labels to train a noise detector. To handle clean data streams, we further inject Gaussian noise during adaptation, preventing the detector from misclassifying clean samples as noisy. Beyond the ZS-NTTA, AdaND can also improve the zero-shot out-of-distribution (ZS-OOD) detection ability of VLMs. Experiments show that AdaND outperforms in both ZS-NTTA and ZS-OOD detection. On ImageNet, AdaND achieves a notable improvement of $8.32\%$ in harmonic mean accuracy ($\text{Acc}_\text{H}$) for ZS-NTTA and $9.40\%$ in FPR95 for ZS-OOD detection, compared to SOTA methods. Importantly, AdaND is computationally efficient and comparable to the model-frozen method. The code is publicly available at: https://github.com/tmlr-group/ZS-NTTA.

Related papers

• Unsupervised CP-UNet Framework for Denoising DAS Data with Decay Noise [13.466125373185399]
  Distributed acoustic sensor (DAS) technology leverages optical fiber cables to detect acoustic signals. DAS exhibits a lower signal-to-noise ratio (S/N) compared to geophones. This reduced S/N can negatively impact data analyses containing inversion and interpretation.
  arXiv  Detail & Related papers  (2025-02-19T03:09:49Z)
• An accurate detection is not all you need to combat label noise in web-noisy datasets [23.020126612431746]
  We show that direct estimation of the separating hyperplane can indeed offer an accurate detection of OOD samples. We propose a hybrid solution that alternates between noise detection using linear separation and a state-of-the-art (SOTA) small-loss approach.
  arXiv  Detail & Related papers  (2024-07-08T00:21:42Z)
• Efficient Test-Time Adaptation of Vision-Language Models [58.3646257833533]
  Test-time adaptation with pre-trained vision-language models has attracted increasing attention for tackling distribution shifts during the test time. We design TDA, a training-free dynamic adapter that enables effective and efficient test-time adaptation with vision-language models.
  arXiv  Detail & Related papers  (2024-03-27T06:37:51Z)
• Robust Tiny Object Detection in Aerial Images amidst Label Noise [50.257696872021164]
  This study addresses the issue of tiny object detection under noisy label supervision. We propose a DeNoising Tiny Object Detector (DN-TOD), which incorporates a Class-aware Label Correction scheme. Our method can be seamlessly integrated into both one-stage and two-stage object detection pipelines.
  arXiv  Detail & Related papers  (2024-01-16T02:14:33Z)
• SoTTA: Robust Test-Time Adaptation on Noisy Data Streams [9.490557638324084]
  Test-time adaptation (TTA) aims to address distributional shifts between training and testing data. Most TTA methods assume benign test streams, while test samples could be unexpectedly diverse in the wild. We present Screening-out Test-Time Adaptation (SoTTA), a novel TTA algorithm that is robust to noisy samples.
  arXiv  Detail & Related papers  (2023-10-16T05:15:35Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• DDPM-CD: Denoising Diffusion Probabilistic Models as Feature Extractors for Change Detection [31.125812018296127]
  We introduce a novel approach for change detection by pre-training a Deno Diffusionising Probabilistic Model (DDPM) DDPM learns the training data distribution by gradually converting training images into a Gaussian distribution using a Markov chain. During inference (i.e., sampling), they can generate a diverse set of samples closer to the training distribution. Experiments conducted on the LEVIR-CD, WHU-CD, DSIFN-CD, and CDD datasets demonstrate that the proposed DDPM-CD method significantly outperforms the existing change detection methods in terms of F1 score, I
  arXiv  Detail & Related papers  (2022-06-23T17:58:29Z)
• Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
  Test-time adaptation seeks to tackle potential distribution shifts between training and testing data. We propose an active sample selection criterion to identify reliable and non-redundant samples. We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
  arXiv  Detail & Related papers  (2022-04-06T06:39:40Z)
• DAAIN: Detection of Anomalous and Adversarial Input using Normalizing Flows [52.31831255787147]
  We introduce a novel technique, DAAIN, to detect out-of-distribution (OOD) inputs and adversarial attacks (AA) Our approach monitors the inner workings of a neural network and learns a density estimator of the activation distribution. Our model can be trained on a single GPU making it compute efficient and deployable without requiring specialized accelerators.
  arXiv  Detail & Related papers  (2021-05-30T22:07:13Z)
• NAT: Noise-Aware Training for Robust Neural Sequence Labeling [30.91638109413785]
  We propose two Noise-Aware Training (NAT) objectives that improve robustness of sequence labeling performed on input. Our data augmentation method trains a neural model using a mixture of clean and noisy samples, whereas our stability training algorithm encourages the model to create a noise-invariant latent representation. Experiments on English and German named entity recognition benchmarks confirmed that NAT consistently improved robustness of popular sequence labeling models.
  arXiv  Detail & Related papers  (2020-05-14T17:30:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.