Subspace Adversarial Training
- URL: http://arxiv.org/abs/2111.12229v1
- Date: Wed, 24 Nov 2021 02:18:37 GMT
- Title: Subspace Adversarial Training
- Authors: Tao Li, Yingwen Wu, Sizhe Chen, Kun Fang, Xiaolin Huang
- Abstract summary: We propose a new AT method, subspace adversarial training (Sub-AT), which constrains the AT in a carefully extracted subspace.
In subspace, we also allow single-step AT with larger steps and larger radius, which further improves the robustness performance.
Our pure single-step AT can reach over $mathbf51%$ robust accuracy against strong PGD-50 attack with radius $8/255$ on CIFAR-10.
- Score: 24.47599337641455
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Single-step adversarial training (AT) has received wide attention as it
proved to be both efficient and robust. However, a serious problem of
catastrophic overfitting exists, i.e., the robust accuracy against projected
gradient descent (PGD) attack suddenly drops to $0\%$ during the training. In
this paper, we understand this problem from a novel perspective of optimization
and firstly reveal the close link between the fast-growing gradient of each
sample and overfitting, which can also be applied to understand the robust
overfitting phenomenon in multi-step AT. To control the growth of the gradient
during the training, we propose a new AT method, subspace adversarial training
(Sub-AT), which constrains the AT in a carefully extracted subspace. It
successfully resolves both two kinds of overfitting and hence significantly
boosts the robustness. In subspace, we also allow single-step AT with larger
steps and larger radius, which further improves the robustness performance. As
a result, we achieve the state-of-the-art single-step AT performance: our pure
single-step AT can reach over $\mathbf{51}\%$ robust accuracy against strong
PGD-50 attack with radius $8/255$ on CIFAR-10, even surpassing the standard
multi-step PGD-10 AT with huge computational advantages. The code is
released$\footnote{\url{https://github.com/nblt/Sub-AT}}$.
Related papers
- RAMP: Boosting Adversarial Robustness Against Multiple $l_p$ Perturbations for Universal Robustness [4.188296977882316]
We propose a novel training framework textbfRAMP, to boost the robustness against multiple $l_p$ perturbations.
For training from scratch, textbfRAMP achieves a union accuracy of $44.6%$ and good clean accuracy of $81.2%$ on ResNet-18 against AutoAttack on CIFAR-10.
arXiv Detail & Related papers (2024-02-09T23:29:54Z) - Bag of Tricks for FGSM Adversarial Training [30.25966570584856]
Adversarial training (AT) with samples generated by Fast Gradient Sign Method (FGSM), also known as FGSM-AT, is a computationally simple method to train robust networks.
During its training procedure, an unstable mode of "catastrophic overfitting" has been identified in arXiv:2001.03994 [cs.LG], where the robust accuracy abruptly drops to zero within a single training step.
In this work, we provide the first study, which thoroughly examines a collection of tricks to overcome the catastrophic overfitting in FGSM-AT.
arXiv Detail & Related papers (2022-09-06T17:53:21Z) - Towards Alternative Techniques for Improving Adversarial Robustness:
Analysis of Adversarial Training at a Spectrum of Perturbations [5.18694590238069]
Adversarial training (AT) and its variants have spearheaded progress in improving neural network robustness to adversarial perturbations.
We focus on models, trained on a spectrum of $epsilon$ values.
We identify alternative improvements to AT that otherwise wouldn't have been apparent at a single $epsilon$.
arXiv Detail & Related papers (2022-06-13T22:01:21Z) - Distributed Adversarial Training to Robustify Deep Neural Networks at
Scale [100.19539096465101]
Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification.
To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training.
We propose a large-batch adversarial training framework implemented over multiple machines.
arXiv Detail & Related papers (2022-06-13T15:39:43Z) - Fast Adversarial Training with Adaptive Step Size [62.37203478589929]
We study the phenomenon from the perspective of training instances.
We propose a simple but effective method, Adversarial Training with Adaptive Step size (ATAS)
ATAS learns an instancewise adaptive step size that is inversely proportional to its gradient norm.
arXiv Detail & Related papers (2022-06-06T08:20:07Z) - Sparsity Winning Twice: Better Robust Generalization from More Efficient
Training [94.92954973680914]
We introduce two alternatives for sparse adversarial training: (i) static sparsity and (ii) dynamic sparsity.
We find both methods to yield win-win: substantially shrinking the robust generalization gap and alleviating the robust overfitting.
Our approaches can be combined with existing regularizers, establishing new state-of-the-art results in adversarial training.
arXiv Detail & Related papers (2022-02-20T15:52:08Z) - Revisiting and Advancing Fast Adversarial Training Through The Lens of
Bi-Level Optimization [60.72410937614299]
We propose a new tractable bi-level optimization problem, design and analyze a new set of algorithms termed Bi-level AT (FAST-BAT)
FAST-BAT is capable of defending sign-based projected descent (PGD) attacks without calling any gradient sign method and explicit robust regularization.
arXiv Detail & Related papers (2021-12-23T06:25:36Z) - Sparse and Imperceptible Adversarial Attack via a Homotopy Algorithm [93.80082636284922]
Sparse adversarial attacks can fool deep networks (DNNs) by only perturbing a few pixels.
Recent efforts combine it with another l_infty perturbation on magnitudes.
We propose a homotopy algorithm to tackle the sparsity and neural perturbation framework.
arXiv Detail & Related papers (2021-06-10T20:11:36Z) - Patch-wise++ Perturbation for Adversarial Targeted Attacks [132.58673733817838]
We propose a patch-wise iterative method (PIM) aimed at crafting adversarial examples with high transferability.
Specifically, we introduce an amplification factor to the step size in each iteration, and one pixel's overall gradient overflowing the $epsilon$-constraint is properly assigned to its surrounding regions.
Compared with the current state-of-the-art attack methods, we significantly improve the success rate by 35.9% for defense models and 32.7% for normally trained models.
arXiv Detail & Related papers (2020-12-31T08:40:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.