Fast Adversarial Training with Adaptive Step Size
- URL: http://arxiv.org/abs/2206.02417v1
- Date: Mon, 6 Jun 2022 08:20:07 GMT
- Title: Fast Adversarial Training with Adaptive Step Size
- Authors: Zhichao Huang, Yanbo Fan, Chen Liu, Weizhong Zhang, Yong Zhang,
Mathieu Salzmann, Sabine S\"usstrunk, Jue Wang
- Abstract summary: We study the phenomenon from the perspective of training instances.
We propose a simple but effective method, Adversarial Training with Adaptive Step size (ATAS)
ATAS learns an instancewise adaptive step size that is inversely proportional to its gradient norm.
- Score: 62.37203478589929
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While adversarial training and its variants have shown to be the most
effective algorithms to defend against adversarial attacks, their extremely
slow training process makes it hard to scale to large datasets like ImageNet.
The key idea of recent works to accelerate adversarial training is to
substitute multi-step attacks (e.g., PGD) with single-step attacks (e.g.,
FGSM). However, these single-step methods suffer from catastrophic overfitting,
where the accuracy against PGD attack suddenly drops to nearly 0% during
training, destroying the robustness of the networks. In this work, we study the
phenomenon from the perspective of training instances. We show that
catastrophic overfitting is instance-dependent and fitting instances with
larger gradient norm is more likely to cause catastrophic overfitting. Based on
our findings, we propose a simple but effective method, Adversarial Training
with Adaptive Step size (ATAS). ATAS learns an instancewise adaptive step size
that is inversely proportional to its gradient norm. The theoretical analysis
shows that ATAS converges faster than the commonly adopted non-adaptive
counterparts. Empirically, ATAS consistently mitigates catastrophic overfitting
and achieves higher robust accuracy on CIFAR10, CIFAR100 and ImageNet when
evaluated on various adversarial budgets.
Related papers
- Detection and Mitigation of Byzantine Attacks in Distributed Training [24.951227624475443]
An abnormal Byzantine behavior of the worker nodes can derail the training and compromise the quality of the inference.
Recent work considers a wide range of attack models and has explored robust aggregation and/or computational redundancy to correct the distorted gradients.
In this work, we consider attack models ranging from strong ones: $q$ omniscient adversaries with full knowledge of the defense protocol that can change from iteration to iteration to weak ones: $q$ randomly chosen adversaries with limited collusion abilities.
arXiv Detail & Related papers (2022-08-17T05:49:52Z) - Versatile Weight Attack via Flipping Limited Bits [68.45224286690932]
We study a novel attack paradigm, which modifies model parameters in the deployment stage.
Considering the effectiveness and stealthiness goals, we provide a general formulation to perform the bit-flip based weight attack.
We present two cases of the general formulation with different malicious purposes, i.e., single sample attack (SSA) and triggered samples attack (TSA)
arXiv Detail & Related papers (2022-07-25T03:24:58Z) - Prior-Guided Adversarial Initialization for Fast Adversarial Training [84.56377396106447]
We investigate the difference between the training processes of adversarial examples (AEs) of Fast adversarial training (FAT) and standard adversarial training (SAT)
We observe that the attack success rate of adversarial examples (AEs) of FAT gets worse gradually in the late training stage, resulting in overfitting.
Based on the observation, we propose a prior-guided FGSM initialization method to avoid overfitting.
The proposed method can prevent catastrophic overfitting and outperform state-of-the-art FAT methods.
arXiv Detail & Related papers (2022-07-18T18:13:10Z) - Distributed Adversarial Training to Robustify Deep Neural Networks at
Scale [100.19539096465101]
Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification.
To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training.
We propose a large-batch adversarial training framework implemented over multiple machines.
arXiv Detail & Related papers (2022-06-13T15:39:43Z) - Revisiting and Advancing Fast Adversarial Training Through The Lens of
Bi-Level Optimization [60.72410937614299]
We propose a new tractable bi-level optimization problem, design and analyze a new set of algorithms termed Bi-level AT (FAST-BAT)
FAST-BAT is capable of defending sign-based projected descent (PGD) attacks without calling any gradient sign method and explicit robust regularization.
arXiv Detail & Related papers (2021-12-23T06:25:36Z) - Adaptive Feature Alignment for Adversarial Training [56.17654691470554]
CNNs are typically vulnerable to adversarial attacks, which pose a threat to security-sensitive applications.
We propose the adaptive feature alignment (AFA) to generate features of arbitrary attacking strengths.
Our method is trained to automatically align features of arbitrary attacking strength.
arXiv Detail & Related papers (2021-05-31T17:01:05Z) - Lagrangian Objective Function Leads to Improved Unforeseen Attack
Generalization in Adversarial Training [0.0]
Adversarial training (AT) has been shown effective to reach a robust model against the attack that is used during training.
We propose a simple modification to the AT that mitigates the mentioned issue.
We show that our attack is faster than other attack schemes that are designed for unseen attack generalization.
arXiv Detail & Related papers (2021-03-29T07:23:46Z) - Robust Single-step Adversarial Training with Regularizer [11.35007968593652]
We propose a novel Fast Gradient Sign Method with PGD Regularization (FGSMPR) to boost the efficiency of adversarial training without catastrophic overfitting.
Experiments demonstrate that our proposed method can train a robust deep network for L$_infty$-perturbations with FGSM adversarial training.
arXiv Detail & Related papers (2021-02-05T19:07:10Z) - Understanding Catastrophic Overfitting in Single-step Adversarial
Training [9.560980936110234]
"catastrophic overfitting" is a phenomenon in which the robust accuracy against projected gradient descent suddenly decreases to 0% after a few epochs.
We propose a simple method that not only prevents catastrophic overfitting, but also overrides the belief that it is difficult to prevent multi-step adversarial attacks with single-step adversarial training.
arXiv Detail & Related papers (2020-10-05T06:13:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.