Prior-Guided Adversarial Initialization for Fast Adversarial Training

• URL: http://arxiv.org/abs/2207.08859v1
• Date: Mon, 18 Jul 2022 18:13:10 GMT
• Title: Prior-Guided Adversarial Initialization for Fast Adversarial Training
• Authors: Xiaojun Jia, Yong Zhang, Xingxing Wei, Baoyuan Wu, Ke Ma, Jue Wang, Xiaochun Cao
• Abstract summary: We investigate the difference between the training processes of adversarial examples (AEs) of Fast adversarial training (FAT) and standard adversarial training (SAT) We observe that the attack success rate of adversarial examples (AEs) of FAT gets worse gradually in the late training stage, resulting in overfitting. Based on the observation, we propose a prior-guided FGSM initialization method to avoid overfitting. The proposed method can prevent catastrophic overfitting and outperform state-of-the-art FAT methods.
• Score: 84.56377396106447
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fast adversarial training (FAT) effectively improves the efficiency of standard adversarial training (SAT). However, initial FAT encounters catastrophic overfitting, i.e.,the robust accuracy against adversarial attacks suddenly and dramatically decreases. Though several FAT variants spare no effort to prevent overfitting, they sacrifice much calculation cost. In this paper, we explore the difference between the training processes of SAT and FAT and observe that the attack success rate of adversarial examples (AEs) of FAT gets worse gradually in the late training stage, resulting in overfitting. The AEs are generated by the fast gradient sign method (FGSM) with a zero or random initialization. Based on the observation, we propose a prior-guided FGSM initialization method to avoid overfitting after investigating several initialization strategies, improving the quality of the AEs during the whole training process. The initialization is formed by leveraging historically generated AEs without additional calculation cost. We further provide a theoretical analysis for the proposed initialization method. We also propose a simple yet effective regularizer based on the prior-guided initialization,i.e., the currently generated perturbation should not deviate too much from the prior-guided initialization. The regularizer adopts both historical and current adversarial perturbations to guide the model learning. Evaluations on four datasets demonstrate that the proposed method can prevent catastrophic overfitting and outperform state-of-the-art FAT methods. The code is released at https://github.com/jiaxiaojunQAQ/FGSM-PGI.

Related papers

• Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective [61.38753850236804]
  Fast adversarial training (FAT) is presented for efficient training and has become a hot research topic. FAT suffers from catastrophic overfitting, which leads to a performance drop compared with multi-step adversarial training. We present an example taxonomy in FAT, which identifies that catastrophic overfitting is caused by the imbalance between the inner and outer optimization in FAT.
  arXiv  Detail & Related papers  (2024-07-22T03:56:27Z)
• Improving Fast Adversarial Training with Prior-Guided Knowledge [80.52575209189365]
  We investigate the relationship between adversarial example quality and catastrophic overfitting by comparing the training processes of standard adversarial training and Fast adversarial training. We find that catastrophic overfitting occurs when the attack success rate of adversarial examples becomes worse.
  arXiv  Detail & Related papers  (2023-04-01T02:18:12Z)
• Bag of Tricks for FGSM Adversarial Training [30.25966570584856]
  Adversarial training (AT) with samples generated by Fast Gradient Sign Method (FGSM), also known as FGSM-AT, is a computationally simple method to train robust networks. During its training procedure, an unstable mode of "catastrophic overfitting" has been identified in arXiv:2001.03994 [cs.LG], where the robust accuracy abruptly drops to zero within a single training step. In this work, we provide the first study, which thoroughly examines a collection of tricks to overcome the catastrophic overfitting in FGSM-AT.
  arXiv  Detail & Related papers  (2022-09-06T17:53:21Z)
• Fast Adversarial Training with Adaptive Step Size [62.37203478589929]
  We study the phenomenon from the perspective of training instances. We propose a simple but effective method, Adversarial Training with Adaptive Step size (ATAS) ATAS learns an instancewise adaptive step size that is inversely proportional to its gradient norm.
  arXiv  Detail & Related papers  (2022-06-06T08:20:07Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)
• Boosting Fast Adversarial Training with Learnable Adversarial Initialization [79.90495058040537]
  Adrial training (AT) has been demonstrated to be effective in improving model robustness by leveraging adversarial examples for training. To boost training efficiency, fast gradient sign method (FGSM) is adopted in fast AT methods by calculating gradient only once.
  arXiv  Detail & Related papers  (2021-10-11T05:37:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.