On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning

• URL: http://arxiv.org/abs/2410.04682v2
• Date: Tue, 15 Oct 2024 06:27:41 GMT
• Title: On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning
• Authors: Yongyi Su, Yushu Li, Nanqing Liu, Kui Jia, Xulei Yang, Chuan-Sheng Foo, Xun Xu,
• Abstract summary: Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. Existing studies have shown that when TTA is updated with crafted adversarial test samples, the performance on benign samples can deteriorate. We propose an effective and realistic attack method that better produces poisoned samples without access to benign samples.
• Score: 49.17494657762375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. However, this practice exposes TTA to adversarial risks. Existing studies have shown that when TTA is updated with crafted adversarial test samples, also known as test-time poisoned data, the performance on benign samples can deteriorate. Nonetheless, the perceived adversarial risk may be overstated if the poisoned data is generated under overly strong assumptions. In this work, we first review realistic assumptions for test-time data poisoning, including white-box versus grey-box attacks, access to benign data, attack budget, and more. We then propose an effective and realistic attack method that better produces poisoned samples without access to benign samples, and derive an effective in-distribution attack objective. We also design two TTA-aware attack objectives. Our benchmarks of existing attack methods reveal that the TTA methods are more robust than previously believed. In addition, we analyze effective defense strategies to help develop adversarially robust TTA methods.

Related papers

• Poison is Not Traceless: Fully-Agnostic Detection of Poisoning Attacks [4.064462548421468]
  This paper presents a novel fully-agnostic framework, DIVA, that detects attacks solely relying on analyzing the potentially poisoned data set. For evaluation purposes, in this paper, we test DIVA on label-flipping attacks.
  arXiv  Detail & Related papers  (2023-10-24T22:27:44Z)
• Confidence-driven Sampling for Backdoor Attacks [49.72680157684523]
  Backdoor attacks aim to surreptitiously insert malicious triggers into DNN models, granting unauthorized control during testing scenarios. Existing methods lack robustness against defense strategies and predominantly focus on enhancing trigger stealthiness while randomly selecting poisoned samples. We introduce a straightforward yet highly effective sampling methodology that leverages confidence scores. Specifically, it selects samples with lower confidence scores, significantly increasing the challenge for defenders in identifying and countering these attacks.
  arXiv  Detail & Related papers  (2023-10-08T18:57:36Z)
• HINT: Healthy Influential-Noise based Training to Defend against Data Poisoning Attacks [12.929357709840975]
  We propose an efficient and robust training approach to defend against data poisoning attacks based on influence functions. Using influence functions, we craft healthy noise that helps to harden the classification model against poisoning attacks. Our empirical results show that HINT can efficiently protect deep learning models against the effect of both untargeted and targeted poisoning attacks.
  arXiv  Detail & Related papers  (2023-09-15T17:12:19Z)
• On Practical Aspects of Aggregation Defenses against Data Poisoning Attacks [58.718697580177356]
  Attacks on deep learning models with malicious training samples are known as data poisoning. Recent advances in defense strategies against data poisoning have highlighted the effectiveness of aggregation schemes in achieving certified poisoning robustness. Here we focus on Deep Partition Aggregation, a representative aggregation defense, and assess its practical aspects, including efficiency, performance, and robustness.
  arXiv  Detail & Related papers  (2023-06-28T17:59:35Z)
• Temporal Robustness against Data Poisoning [69.01705108817785]
  Data poisoning considers cases when an adversary manipulates the behavior of machine learning algorithms through malicious training data. We propose a temporal threat model of data poisoning with two novel metrics, earliness and duration, which respectively measure how long an attack started in advance and how long an attack lasted.
  arXiv  Detail & Related papers  (2023-02-07T18:59:19Z)
• Uncovering Adversarial Risks of Test-Time Adaptation [41.19226800089764]
  Test-time adaptation (TTA) has been proposed as a promising solution for addressing distribution shifts. We uncover a novel security vulnerability of TTA based on the insight that predictions on benign samples can be impacted by malicious samples in the same batch. We propose Distribution Invading Attack (DIA), which injects a small fraction of malicious data into the test batch.
  arXiv  Detail & Related papers  (2023-01-29T22:58:05Z)
• Adversarial Attacks and Defense for Non-Parametric Two-Sample Tests [73.32304304788838]
  This paper systematically uncovers the failure mode of non-parametric TSTs through adversarial attacks. To enable TST-agnostic attacks, we propose an ensemble attack framework that jointly minimizes the different types of test criteria. To robustify TSTs, we propose a max-min optimization that iteratively generates adversarial pairs to train the deep kernels.
  arXiv  Detail & Related papers  (2022-02-07T11:18:04Z)
• Defending Regression Learners Against Poisoning Attacks [25.06658793731661]
  We introduce a novel Local Intrinsic Dimensionality (LID) based measure called N-LID that measures the local deviation of a given data point's LID with respect to its neighbors. N-LID can distinguish poisoned samples from normal samples and propose an N-LID based defense approach that makes no assumptions of the attacker. We show that the proposed defense mechanism outperforms the state of the art defenses in terms of prediction accuracy (up to 76% lower MSE compared to an undefended ridge model) and running time.
  arXiv  Detail & Related papers  (2020-08-21T03:02:58Z)
• Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks [74.88735178536159]
  Data poisoning is the number one concern among threats ranging from model stealing to adversarial attacks. We observe that data poisoning and backdoor attacks are highly sensitive to variations in the testing setup. We apply rigorous tests to determine the extent to which we should fear them.
  arXiv  Detail & Related papers  (2020-06-22T18:34:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.