Efficient Adversarial Training with Robust Early-Bird Tickets

• URL: http://arxiv.org/abs/2211.07263v2
• Date: Tue, 15 Nov 2022 01:34:39 GMT
• Title: Efficient Adversarial Training with Robust Early-Bird Tickets
• Authors: Zhiheng Xi, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang
• Abstract summary: We find that robust connectivity patterns emerge in the early training phase, far before parameters converge. Inspired by this finding, we dig out robust early-bird tickets to develop an efficient adversarial training method. Experiments show that the proposed efficient adversarial training method can achieve up to $7times sim 13 times$ training speedups.
• Score: 57.72115485770303
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial training is one of the most powerful methods to improve the robustness of pre-trained language models (PLMs). However, this approach is typically more expensive than traditional fine-tuning because of the necessity to generate adversarial examples via gradient descent. Delving into the optimization process of adversarial training, we find that robust connectivity patterns emerge in the early training phase (typically $0.15\sim0.3$ epochs), far before parameters converge. Inspired by this finding, we dig out robust early-bird tickets (i.e., subnetworks) to develop an efficient adversarial training method: (1) searching for robust tickets with structured sparsity in the early stage; (2) fine-tuning robust tickets in the remaining time. To extract the robust tickets as early as possible, we design a ticket convergence metric to automatically terminate the searching process. Experiments show that the proposed efficient adversarial training method can achieve up to $7\times \sim 13 \times$ training speedups while maintaining comparable or even better robustness compared to the most competitive state-of-the-art adversarial training methods.

Related papers

• Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averaging [73.78965374696608]
  We study over 10 fast adversarial training methods in terms of adversarial robustness and training costs. We revisit the effectiveness and efficiency of fast adversarial training techniques in preventing Catastrophic Overfitting. We propose a FGSM-based fast adversarial training method equipped with Lipschitz regularization and Auto Weight averaging.
  arXiv  Detail & Related papers  (2023-08-22T13:50:49Z)
• Adversarial Coreset Selection for Efficient Robust Training [11.510009152620666]
  We show how selecting a small subset of training data provides a principled approach to reducing the time complexity of robust training. We conduct extensive experiments to demonstrate that our approach speeds up adversarial training by 2-3 times.
  arXiv  Detail & Related papers  (2022-09-13T07:37:53Z)
• $\ell_\infty$-Robustness and Beyond: Unleashing Efficient Adversarial Training [11.241749205970253]
  We show how selecting a small subset of training data provides a more principled approach towards reducing the time complexity of robust training. Our approach speeds up adversarial training by 2-3 times, while experiencing a small reduction in the clean and robust accuracy.
  arXiv  Detail & Related papers  (2021-12-01T09:55:01Z)
• Boosting Fast Adversarial Training with Learnable Adversarial Initialization [79.90495058040537]
  Adrial training (AT) has been demonstrated to be effective in improving model robustness by leveraging adversarial examples for training. To boost training efficiency, fast gradient sign method (FGSM) is adopted in fast AT methods by calculating gradient only once.
  arXiv  Detail & Related papers  (2021-10-11T05:37:00Z)
• Efficient Robust Training via Backward Smoothing [125.91185167854262]
  Adversarial training is the most effective strategy in defending against adversarial examples. It suffers from high computational costs due to the iterative adversarial attacks in each training step. Recent studies show that it is possible to achieve fast Adversarial Training by performing a single-step attack.
  arXiv  Detail & Related papers  (2020-10-03T04:37:33Z)
• Towards Understanding Fast Adversarial Training [91.8060431517248]
  We conduct experiments to understand the behavior of fast adversarial training. We show the key to its success is the ability to recover from overfitting to weak attacks.
  arXiv  Detail & Related papers  (2020-06-04T18:19:43Z)
• Fast is better than free: Revisiting adversarial training [86.11788847990783]
  We show that it is possible to train empirically robust models using a much weaker and cheaper adversary. We identify a failure mode referred to as "catastrophic overfitting" which may have caused previous attempts to use FGSM adversarial training to fail.
  arXiv  Detail & Related papers  (2020-01-12T20:30:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.