Multi-stage Optimization based Adversarial Training

• URL: http://arxiv.org/abs/2106.15357v1
• Date: Sat, 26 Jun 2021 07:59:52 GMT
• Title: Multi-stage Optimization based Adversarial Training
• Authors: Xiaosen Wang, Chuanbiao Song, Liwei Wang, Kun He
• Abstract summary: We propose a Multi-stage Optimization based Adversarial Training (MOAT) method that periodically trains the model on mixed benign examples. Under similar amount of training overhead, the proposed MOAT exhibits better robustness than either single-step or multi-step adversarial training methods.
• Score: 16.295921205749934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the field of adversarial robustness, there is a common practice that adopts the single-step adversarial training for quickly developing adversarially robust models. However, the single-step adversarial training is most likely to cause catastrophic overfitting, as after a few training epochs it will be hard to generate strong adversarial examples to continuously boost the adversarial robustness. In this work, we aim to avoid the catastrophic overfitting by introducing multi-step adversarial examples during the single-step adversarial training. Then, to balance the large training overhead of generating multi-step adversarial examples, we propose a Multi-stage Optimization based Adversarial Training (MOAT) method that periodically trains the model on mixed benign examples, single-step adversarial examples, and multi-step adversarial examples stage by stage. In this way, the overall training overhead is reduced significantly, meanwhile, the model could avoid catastrophic overfitting. Extensive experiments on CIFAR-10 and CIFAR-100 datasets demonstrate that under similar amount of training overhead, the proposed MOAT exhibits better robustness than either single-step or multi-step adversarial training methods.

Related papers

• Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworks [69.54774045493227]
  A drawback of adversarial training is the computational overhead introduced by the generation of adversarial examples. We propose to exploit the interior building blocks of the model to improve efficiency. Compared with previous methods, our method not only reduces the training cost but also achieves better model robustness.
  arXiv  Detail & Related papers  (2023-10-24T01:36:20Z)
• On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training [72.95029777394186]
  Adversarial training is a popular method to robustify models against adversarial attacks. We investigate this phenomenon from the perspective of training instances. We show that the decay in generalization performance of adversarial training is a result of the model's attempt to fit hard adversarial instances.
  arXiv  Detail & Related papers  (2021-12-14T12:19:24Z)
• Robust Single-step Adversarial Training with Regularizer [11.35007968593652]
  We propose a novel Fast Gradient Sign Method with PGD Regularization (FGSMPR) to boost the efficiency of adversarial training without catastrophic overfitting. Experiments demonstrate that our proposed method can train a robust deep network for L$_infty$-perturbations with FGSM adversarial training.
  arXiv  Detail & Related papers  (2021-02-05T19:07:10Z)
• Self-Progressing Robust Training [146.8337017922058]
  Current robust training methods such as adversarial training explicitly uses an "attack" to generate adversarial examples. We propose a new framework called SPROUT, self-progressing robust training. Our results shed new light on scalable, effective and attack-independent robust training methods.
  arXiv  Detail & Related papers  (2020-12-22T00:45:24Z)
• Adversarial Training with Stochastic Weight Average [4.633908654744751]
  Adrial training deep neural networks often experience serious overfitting problem. In traditional machine learning, one way to relieve overfitting from the lack of data is to use ensemble methods. In this paper, we propose adversarial training with weight average (SWA) While performing adversarial training, we aggregate the temporal weight states in the trajectory of training.
  arXiv  Detail & Related papers  (2020-09-21T04:47:20Z)
• Single-step Adversarial training with Dropout Scheduling [59.50324605982158]
  We show that models trained using single-step adversarial training method learn to prevent the generation of single-step adversaries. Models trained using proposed single-step adversarial training method are robust against both single-step and multi-step adversarial attacks.
  arXiv  Detail & Related papers  (2020-04-18T14:14:00Z)
• Regularizers for Single-step Adversarial Training [49.65499307547198]
  We propose three types of regularizers that help to learn robust models using single-step adversarial training methods. Regularizers mitigate the effect of gradient masking by harnessing on properties that differentiate a robust model from that of a pseudo robust model.
  arXiv  Detail & Related papers  (2020-02-03T09:21:04Z)
• Efficient Adversarial Training with Transferable Adversarial Examples [58.62766224452761]
  We show that there is high transferability between models from neighboring epochs in the same training process. We propose a novel method, Adversarial Training with Transferable Adversarial Examples (ATTA) that can enhance the robustness of trained models.
  arXiv  Detail & Related papers  (2019-12-27T03:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.