Improving the affordability of robustness training for DNNs

• URL: http://arxiv.org/abs/2002.04237v2
• Date: Thu, 30 Apr 2020 04:17:09 GMT
• Title: Improving the affordability of robustness training for DNNs
• Authors: Sidharth Gupta, Parijat Dube, Ashish Verma
• Abstract summary: We show that the initial phase of adversarial training is redundant and can be replaced with natural training which significantly improves the computational efficiency. We show that our proposed method can reduce the training time by a factor of up to 2.5 with comparable or better model test accuracy and generalization on various strengths of adversarial attacks.
• Score: 11.971637253035107
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Projected Gradient Descent (PGD) based adversarial training has become one of the most prominent methods for building robust deep neural network models. However, the computational complexity associated with this approach, due to the maximization of the loss function when finding adversaries, is a longstanding problem and may be prohibitive when using larger and more complex models. In this paper we show that the initial phase of adversarial training is redundant and can be replaced with natural training which significantly improves the computational efficiency. We demonstrate that this efficiency gain can be achieved without any loss in accuracy on natural and adversarial test samples. We support our argument with insights on the nature of the adversaries and their relative strength during the training process. We show that our proposed method can reduce the training time by a factor of up to 2.5 with comparable or better model test accuracy and generalization on various strengths of adversarial attacks.

Related papers

• Efficient Adversarial Training in LLMs with Continuous Attacks [99.5882845458567]
  Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses. C-AdvIPO is an adversarial variant of IPO that does not require utility data for adversarially robust alignment.
  arXiv  Detail & Related papers  (2024-05-24T14:20:09Z)
• Reducing Adversarial Training Cost with Gradient Approximation [0.3916094706589679]
  We propose a new and efficient adversarial training method, adversarial training with gradient approximation (GAAT) to reduce the cost of building up robust models. Our proposed method saves up to 60% of the training time with comparable model test accuracy on datasets.
  arXiv  Detail & Related papers  (2023-09-18T03:55:41Z)
• Outlier Robust Adversarial Training [57.06824365801612]
  We introduce Outlier Robust Adversarial Training (ORAT) in this work. ORAT is based on a bi-level optimization formulation of adversarial training with a robust rank-based loss function. We show that the learning objective of ORAT satisfies the $mathcalH$-consistency in binary classification, which establishes it as a proper surrogate to adversarial 0/1 loss.
  arXiv  Detail & Related papers  (2023-09-10T21:36:38Z)
• A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking [54.89987482509155]
  robustness of deep neural networks is usually lacking under adversarial examples, common corruptions, and distribution shifts. We establish a comprehensive benchmark robustness called textbfARES-Bench on the image classification task. By designing the training settings accordingly, we achieve the new state-of-the-art adversarial robustness.
  arXiv  Detail & Related papers  (2023-02-28T04:26:20Z)
• Enhancing Adversarial Training with Feature Separability [52.39305978984573]
  We introduce a new concept of adversarial training graph (ATG) with which the proposed adversarial training with feature separability (ATFS) enables to boost the intra-class feature similarity and increase inter-class feature variance. Through comprehensive experiments, we demonstrate that the proposed ATFS framework significantly improves both clean and robust performance.
  arXiv  Detail & Related papers  (2022-05-02T04:04:23Z)
• Gradient-Guided Dynamic Efficient Adversarial Training [6.980357450216633]
  Adversarial training is arguably an effective but time-consuming way to train robust deep neural networks that can withstand strong adversarial attacks. We propose the Dynamic Efficient Adversarial Training (DEAT), which gradually increases the adversarial iteration during training.
  arXiv  Detail & Related papers  (2021-03-04T14:57:53Z)
• Robust Single-step Adversarial Training with Regularizer [11.35007968593652]
  We propose a novel Fast Gradient Sign Method with PGD Regularization (FGSMPR) to boost the efficiency of adversarial training without catastrophic overfitting. Experiments demonstrate that our proposed method can train a robust deep network for L$_infty$-perturbations with FGSM adversarial training.
  arXiv  Detail & Related papers  (2021-02-05T19:07:10Z)
• A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning [90.44219200633286]
  We propose a simple yet very effective adversarial fine-tuning approach based on a $textitslow start, fast decay$ learning rate scheduling strategy. Experimental results show that the proposed adversarial fine-tuning approach outperforms the state-of-the-art methods on CIFAR-10, CIFAR-100 and ImageNet datasets.
  arXiv  Detail & Related papers  (2020-12-25T20:50:15Z)
• On the Generalization Properties of Adversarial Training [21.79888306754263]
  This paper studies the generalization performance of a generic adversarial training algorithm. A series of numerical studies are conducted to demonstrate how the smoothness and L1 penalization help improve the adversarial robustness of models.
  arXiv  Detail & Related papers  (2020-08-15T02:32:09Z)
• Towards Understanding Fast Adversarial Training [91.8060431517248]
  We conduct experiments to understand the behavior of fast adversarial training. We show the key to its success is the ability to recover from overfitting to weak attacks.
  arXiv  Detail & Related papers  (2020-06-04T18:19:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.