Improving Fast Adversarial Training via Self-Knowledge Guidance

• URL: http://arxiv.org/abs/2409.17589v1
• Date: Thu, 26 Sep 2024 07:12:04 GMT
• Title: Improving Fast Adversarial Training via Self-Knowledge Guidance
• Authors: Chengze Jiang, Junkai Wang, Minjing Dong, Jie Gui, Xinli Shi, Yuan Cao, Yuan Yan Tang, James Tin-Yau Kwok,
• Abstract summary: We conduct a comprehensive study of the imbalance issue in fast adversarial training (FAT) We observe an obvious class disparity regarding their performances. This disparity could be embodied from a perspective of alignment between clean and robust accuracy.
• Score: 30.299641184202972
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adversarial training has achieved remarkable advancements in defending against adversarial attacks. Among them, fast adversarial training (FAT) is gaining attention for its ability to achieve competitive robustness with fewer computing resources. Existing FAT methods typically employ a uniform strategy that optimizes all training data equally without considering the influence of different examples, which leads to an imbalanced optimization. However, this imbalance remains unexplored in the field of FAT. In this paper, we conduct a comprehensive study of the imbalance issue in FAT and observe an obvious class disparity regarding their performances. This disparity could be embodied from a perspective of alignment between clean and robust accuracy. Based on the analysis, we mainly attribute the observed misalignment and disparity to the imbalanced optimization in FAT, which motivates us to optimize different training data adaptively to enhance robustness. Specifically, we take disparity and misalignment into consideration. First, we introduce self-knowledge guided regularization, which assigns differentiated regularization weights to each class based on its training state, alleviating class disparity. Additionally, we propose self-knowledge guided label relaxation, which adjusts label relaxation according to the training accuracy, alleviating the misalignment and improving robustness. By combining these methods, we formulate the Self-Knowledge Guided FAT (SKG-FAT), leveraging naturally generated knowledge during training to enhance the adversarial robustness without compromising training efficiency. Extensive experiments on four standard datasets demonstrate that the SKG-FAT improves the robustness and preserves competitive clean accuracy, outperforming the state-of-the-art methods.

Related papers

• Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective [61.38753850236804]
  Fast adversarial training (FAT) is presented for efficient training and has become a hot research topic. FAT suffers from catastrophic overfitting, which leads to a performance drop compared with multi-step adversarial training. We present an example taxonomy in FAT, which identifies that catastrophic overfitting is caused by the imbalance between the inner and outer optimization in FAT.
  arXiv  Detail & Related papers  (2024-07-22T03:56:27Z)
• Omnipotent Adversarial Training in the Wild [20.239704959690936]
  We propose Omniversapotent Adrial Training (OAT) strategy to train a model on an imbalanced and noisy dataset. OAT consists of two innovative methodologies to address the imperfection in the training set. OAT outperforms other baselines by more than 20% clean accuracy improvement and 10% robust accuracy improvement.
  arXiv  Detail & Related papers  (2023-07-14T07:09:57Z)
• Improving Fast Adversarial Training with Prior-Guided Knowledge [80.52575209189365]
  We investigate the relationship between adversarial example quality and catastrophic overfitting by comparing the training processes of standard adversarial training and Fast adversarial training. We find that catastrophic overfitting occurs when the attack success rate of adversarial examples becomes worse.
  arXiv  Detail & Related papers  (2023-04-01T02:18:12Z)
• CalFAT: Calibrated Federated Adversarial Training with Label Skewness [46.47690793066599]
  We propose a Calibrated FAT (CalFAT) approach to tackle the instability issue by calibrating the logits adaptively to balance the classes. We show both theoretically and empirically that the optimization of CalFAT leads to homogeneous local models across the clients and much improved convergence rate and final performance.
  arXiv  Detail & Related papers  (2022-05-30T08:49:20Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free [115.81899803240758]
  Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. This paper asks how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies. Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework.
  arXiv  Detail & Related papers  (2020-10-22T16:06:34Z)
• Adversarial Robustness on In- and Out-Distribution Improves Explainability [109.68938066821246]
  RATIO is a training procedure for robustness via Adversarial Training on In- and Out-distribution. RATIO achieves state-of-the-art $l$-adrial on CIFAR10 and maintains better clean accuracy.
  arXiv  Detail & Related papers  (2020-03-20T18:57:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.