HYDRA: Pruning Adversarially Robust Neural Networks

• URL: http://arxiv.org/abs/2002.10509v3
• Date: Tue, 10 Nov 2020 15:02:00 GMT
• Title: HYDRA: Pruning Adversarially Robust Neural Networks
• Authors: Vikash Sehwag, Shiqi Wang, Prateek Mittal, Suman Jana
• Abstract summary: Deep learning faces two key challenges: lack of robustness against adversarial attacks and large neural network size. We propose to make pruning techniques aware of the robust training objective and let the training objective guide the search for which connections to prune. We demonstrate that our approach, titled HYDRA, achieves compressed networks with state-of-the-art benign and robust accuracy, simultaneously.
• Score: 58.061681100058316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In safety-critical but computationally resource-constrained applications, deep learning faces two key challenges: lack of robustness against adversarial attacks and large neural network size (often millions of parameters). While the research community has extensively explored the use of robust training and network pruning independently to address one of these challenges, only a few recent works have studied them jointly. However, these works inherit a heuristic pruning strategy that was developed for benign training, which performs poorly when integrated with robust training techniques, including adversarial training and verifiable robust training. To overcome this challenge, we propose to make pruning techniques aware of the robust training objective and let the training objective guide the search for which connections to prune. We realize this insight by formulating the pruning objective as an empirical risk minimization problem which is solved efficiently using SGD. We demonstrate that our approach, titled HYDRA, achieves compressed networks with state-of-the-art benign and robust accuracy, simultaneously. We demonstrate the success of our approach across CIFAR-10, SVHN, and ImageNet dataset with four robust training techniques: iterative adversarial training, randomized smoothing, MixTrain, and CROWN-IBP. We also demonstrate the existence of highly robust sub-networks within non-robust networks. Our code and compressed networks are publicly available at \url{https://github.com/inspire-group/compactness-robustness}.

Related papers

• Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data [38.44734564565478]
  We provide a theoretical understanding of adversarial examples and adversarial training algorithms from the perspective of feature learning theory. We show that the adversarial training method can provably strengthen the robust feature learning and suppress the non-robust feature learning.
  arXiv  Detail & Related papers  (2024-10-11T03:59:49Z)
• A Theoretical Perspective on Subnetwork Contributions to Adversarial Robustness [2.064612766965483]
  This paper investigates how the adversarial robustness of a subnetwork contributes to the robustness of the entire network. Experiments show the ability of a robust subnetwork to promote full-network robustness, and investigate the layer-wise dependencies required for this full-network robustness to be achieved.
  arXiv  Detail & Related papers  (2023-07-07T19:16:59Z)
• A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking [54.89987482509155]
  robustness of deep neural networks is usually lacking under adversarial examples, common corruptions, and distribution shifts. We establish a comprehensive benchmark robustness called textbfARES-Bench on the image classification task. By designing the training settings accordingly, we achieve the new state-of-the-art adversarial robustness.
  arXiv  Detail & Related papers  (2023-02-28T04:26:20Z)
• Quantization-aware Interval Bound Propagation for Training Certifiably Robust Quantized Neural Networks [58.195261590442406]
  We study the problem of training and certifying adversarially robust quantized neural networks (QNNs) Recent work has shown that floating-point neural networks that have been verified to be robust can become vulnerable to adversarial attacks after quantization. We present quantization-aware interval bound propagation (QA-IBP), a novel method for training robust QNNs.
  arXiv  Detail & Related papers  (2022-11-29T13:32:38Z)
• Finding Dynamics Preserving Adversarial Winning Tickets [11.05616199881368]
  Pruning methods have been considered in adversarial context to reduce model capacity and improve adversarial robustness simultaneously in training. Existing adversarial pruning methods generally mimic the classical pruning methods for natural training, which follow the three-stage 'training-pruning-fine-tuning' pipelines. We show empirical evidences that AWT preserves the dynamics of adversarial training and achieve equal performance as dense adversarial training.
  arXiv  Detail & Related papers  (2022-02-14T05:34:24Z)
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks [13.863895853997091]
  Distinct from the popular lottery ticket hypothesis, neither the original dense networks nor the identified RSTs need to be trained. We identify the poor adversarial transferability between RSTs of different sparsity ratios drawn from the same randomly dense network. We propose a Random RST Switch (R2S) technique, which randomly switches between different RSTs as a novel defense method.
  arXiv  Detail & Related papers  (2021-10-26T22:52:56Z)
• Self-Progressing Robust Training [146.8337017922058]
  Current robust training methods such as adversarial training explicitly uses an "attack" to generate adversarial examples. We propose a new framework called SPROUT, self-progressing robust training. Our results shed new light on scalable, effective and attack-independent robust training methods.
  arXiv  Detail & Related papers  (2020-12-22T00:45:24Z)
• Rethinking Clustering for Robustness [56.14672993686335]
  ClusTR is a clustering-based and adversary-free training framework to learn robust models. textitClusTR outperforms adversarially-trained networks by up to $4%$ under strong PGD attacks.
  arXiv  Detail & Related papers  (2020-06-13T16:55:51Z)
• Feature Purification: How Adversarial Training Performs Robust Deep Learning [66.05472746340142]
  We show a principle that we call Feature Purification, where we show one of the causes of the existence of adversarial examples is the accumulation of certain small dense mixtures in the hidden weights during the training process of a neural network. We present both experiments on the CIFAR-10 dataset to illustrate this principle, and a theoretical result proving that for certain natural classification tasks, training a two-layer neural network with ReLU activation using randomly gradient descent indeed this principle.
  arXiv  Detail & Related papers  (2020-05-20T16:56:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.