Extreme Miscalibration and the Illusion of Adversarial Robustness

• URL: http://arxiv.org/abs/2402.17509v3
• Date: Sun, 13 Oct 2024 16:11:30 GMT
• Title: Extreme Miscalibration and the Illusion of Adversarial Robustness
• Authors: Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, George Karypis,
• Abstract summary: Adversarial Training is often used to increase model robustness. We show that this observed gain in robustness is an illusion of robustness (IOR) We urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations.
• Score: 66.29268991629085
• License:
• Abstract: Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify. Adversarial Training (AT) is often used to increase model robustness. However, we have discovered an intriguing phenomenon: deliberately or accidentally miscalibrating models masks gradients in a way that interferes with adversarial attack search methods, giving rise to an apparent increase in robustness. We show that this observed gain in robustness is an illusion of robustness (IOR), and demonstrate how an adversary can perform various forms of test-time temperature calibration to nullify the aforementioned interference and allow the adversarial attack to find adversarial examples. Hence, we urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine. Finally, we show how the temperature can be scaled during \textit{training} to improve genuine robustness.

Related papers

• Adversarial Robustness Overestimation and Instability in TRADES [4.063518154926961]
  TRADES sometimes yields disproportionately high PGD validation accuracy compared to the AutoAttack testing accuracy in the multiclass classification task. This discrepancy highlights a significant overestimation of robustness for these instances, potentially linked to gradient masking.
  arXiv  Detail & Related papers  (2024-10-10T07:32:40Z)
• Exploring the Adversarial Frontier: Quantifying Robustness via Adversarial Hypervolume [17.198794644483026]
  We propose a new metric termed adversarial hypervolume, assessing the robustness of deep learning models comprehensively over a range of perturbation intensities. We adopt a novel training algorithm that enhances adversarial robustness uniformly across various perturbation intensities. This research contributes a new measure of robustness and establishes a standard for assessing benchmarking and the resilience of current and future defensive models against adversarial threats.
  arXiv  Detail & Related papers  (2024-03-08T07:03:18Z)
• Mitigating Feature Gap for Adversarial Robustness by Feature Disentanglement [61.048842737581865]
  Adversarial fine-tuning methods aim to enhance adversarial robustness through fine-tuning the naturally pre-trained model in an adversarial training manner. We propose a disentanglement-based approach to explicitly model and remove the latent features that cause the feature gap. Empirical evaluations on three benchmark datasets demonstrate that our approach surpasses existing adversarial fine-tuning methods and adversarial training baselines.
  arXiv  Detail & Related papers  (2024-01-26T08:38:57Z)
• Robust Deep Learning Models Against Semantic-Preserving Adversarial Attack [3.7264705684737893]
  Deep learning models can be fooled by small $l_p$-norm adversarial perturbations and natural perturbations in terms of attributes. We propose a novel attack mechanism named Semantic-Preserving Adversarial (SPA) attack, which can then be used to enhance adversarial training.
  arXiv  Detail & Related papers  (2023-04-08T08:28:36Z)
• Consistent Valid Physically-Realizable Adversarial Attack against Crowd-flow Prediction Models [4.286570387250455]
  deep learning (DL) models can effectively learn city-wide crowd-flow patterns. DL models have been known to perform poorly on inconspicuous adversarial perturbations.
  arXiv  Detail & Related papers  (2023-03-05T13:30:25Z)
• Improving Adversarial Robustness to Sensitivity and Invariance Attacks with Deep Metric Learning [80.21709045433096]
  A standard method in adversarial robustness assumes a framework to defend against samples crafted by minimally perturbing a sample. We use metric learning to frame adversarial regularization as an optimal transport problem. Our preliminary results indicate that regularizing over invariant perturbations in our framework improves both invariant and sensitivity defense.
  arXiv  Detail & Related papers  (2022-11-04T13:54:02Z)
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition [109.62614226793833]
  The trade-off between robustness and accuracy has been widely studied in the adversarial literature. We find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty.
  arXiv  Detail & Related papers  (2022-02-21T10:36:09Z)
• On the Robustness of Quality Measures for GANs [136.18799984346248]
  This work evaluates the robustness of quality measures of generative models such as Inception Score (IS) and Fr'echet Inception Distance (FID) We show that such metrics can also be manipulated by additive pixel perturbations.
  arXiv  Detail & Related papers  (2022-01-31T06:43:09Z)
• Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
  We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs. We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
  arXiv  Detail & Related papers  (2021-06-21T21:42:08Z)
• Improving Calibration through the Relationship with Adversarial Robustness [19.384119330332446]
  We study the connection between adversarial robustness and calibration. We propose Adversarial Robustness based Adaptive Labeling (AR-AdaLS) We find that our method, taking the adversarial robustness of the in-distribution data into consideration, leads to better calibration over the model even under distributional shifts.
  arXiv  Detail & Related papers  (2020-06-29T20:56:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.