Fast Adversarial Training with Smooth Convergence
- URL: http://arxiv.org/abs/2308.12857v1
- Date: Thu, 24 Aug 2023 15:28:52 GMT
- Title: Fast Adversarial Training with Smooth Convergence
- Authors: Mengnan Zhao, Lihe Zhang, Yuqiu Kong and Baocai Yin
- Abstract summary: We analyze the training process of prior Fast adversarial training (FAT) work and observe that catastrophic overfitting is accompanied by the appearance of loss convergence outliers.
To obtain a smooth loss convergence process, we propose a novel oscillatory constraint (dubbed ConvergeSmooth) to limit the loss difference between adjacent epochs.
Our proposed methods are attack-agnostic and thus can improve the training stability of various FAT techniques.
- Score: 51.996943482875366
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Fast adversarial training (FAT) is beneficial for improving the adversarial
robustness of neural networks. However, previous FAT work has encountered a
significant issue known as catastrophic overfitting when dealing with large
perturbation budgets, \ie the adversarial robustness of models declines to near
zero during training.
To address this, we analyze the training process of prior FAT work and
observe that catastrophic overfitting is accompanied by the appearance of loss
convergence outliers.
Therefore, we argue a moderately smooth loss convergence process will be a
stable FAT process that solves catastrophic overfitting.
To obtain a smooth loss convergence process, we propose a novel oscillatory
constraint (dubbed ConvergeSmooth) to limit the loss difference between
adjacent epochs. The convergence stride of ConvergeSmooth is introduced to
balance convergence and smoothing. Likewise, we design weight centralization
without introducing additional hyperparameters other than the loss balance
coefficient.
Our proposed methods are attack-agnostic and thus can improve the training
stability of various FAT techniques.
Extensive experiments on popular datasets show that the proposed methods
efficiently avoid catastrophic overfitting and outperform all previous FAT
methods. Code is available at \url{https://github.com/FAT-CS/ConvergeSmooth}.
Related papers
- Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective [61.38753850236804]
Fast adversarial training (FAT) is presented for efficient training and has become a hot research topic.
FAT suffers from catastrophic overfitting, which leads to a performance drop compared with multi-step adversarial training.
We present an example taxonomy in FAT, which identifies that catastrophic overfitting is caused by the imbalance between the inner and outer optimization in FAT.
arXiv Detail & Related papers (2024-07-22T03:56:27Z) - Preventing Catastrophic Overfitting in Fast Adversarial Training: A Bi-level Optimization Perspective [20.99874786089634]
Adversarial training (AT) has become an effective defense method against adversarial examples (AEs)
Fast AT (FAT) employs a single-step attack strategy to guide the training process.
FAT methods suffer from the catastrophic overfitting problem.
arXiv Detail & Related papers (2024-07-17T09:53:20Z) - Efficient local linearity regularization to overcome catastrophic
overfitting [59.463867084204566]
Catastrophic overfitting (CO) in single-step adversarial training results in abrupt drops in the adversarial test accuracy (even down to 0%)
We introduce a regularization term, called ELLE, to mitigate CO effectively and efficiently in classical AT evaluations.
arXiv Detail & Related papers (2024-01-21T22:55:26Z) - Improving Fast Adversarial Training with Prior-Guided Knowledge [80.52575209189365]
We investigate the relationship between adversarial example quality and catastrophic overfitting by comparing the training processes of standard adversarial training and Fast adversarial training.
We find that catastrophic overfitting occurs when the attack success rate of adversarial examples becomes worse.
arXiv Detail & Related papers (2023-04-01T02:18:12Z) - Prior-Guided Adversarial Initialization for Fast Adversarial Training [84.56377396106447]
We investigate the difference between the training processes of adversarial examples (AEs) of Fast adversarial training (FAT) and standard adversarial training (SAT)
We observe that the attack success rate of adversarial examples (AEs) of FAT gets worse gradually in the late training stage, resulting in overfitting.
Based on the observation, we propose a prior-guided FGSM initialization method to avoid overfitting.
The proposed method can prevent catastrophic overfitting and outperform state-of-the-art FAT methods.
arXiv Detail & Related papers (2022-07-18T18:13:10Z) - Fast Adversarial Training with Adaptive Step Size [62.37203478589929]
We study the phenomenon from the perspective of training instances.
We propose a simple but effective method, Adversarial Training with Adaptive Step size (ATAS)
ATAS learns an instancewise adaptive step size that is inversely proportional to its gradient norm.
arXiv Detail & Related papers (2022-06-06T08:20:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.