Accelerating Certified Robustness Training via Knowledge Transfer
- URL: http://arxiv.org/abs/2210.14283v1
- Date: Tue, 25 Oct 2022 19:12:28 GMT
- Title: Accelerating Certified Robustness Training via Knowledge Transfer
- Authors: Pratik Vaishnavi, Kevin Eykholt, Amir Rahmati
- Abstract summary: We propose a framework for reducing the computational overhead of any certifiably robust training method through knowledge transfer.
Our experiments on CIFAR-10 show that CRT speeds up certified robustness training by $8 times$ on average across three different architecture generations.
- Score: 3.5934248574481717
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Training deep neural network classifiers that are certifiably robust against
adversarial attacks is critical to ensuring the security and reliability of
AI-controlled systems. Although numerous state-of-the-art certified training
methods have been developed, they are computationally expensive and scale
poorly with respect to both dataset and network complexity. Widespread usage of
certified training is further hindered by the fact that periodic retraining is
necessary to incorporate new data and network improvements. In this paper, we
propose Certified Robustness Transfer (CRT), a general-purpose framework for
reducing the computational overhead of any certifiably robust training method
through knowledge transfer. Given a robust teacher, our framework uses a novel
training loss to transfer the teacher's robustness to the student. We provide
theoretical and empirical validation of CRT. Our experiments on CIFAR-10 show
that CRT speeds up certified robustness training by $8 \times$ on average
across three different architecture generations while achieving comparable
robustness to state-of-the-art methods. We also show that CRT can scale to
large-scale datasets like ImageNet.
Related papers
- Training Overhead Ratio: A Practical Reliability Metric for Large Language Model Training Systems [13.880001659156926]
Large Language Models (LLMs) are revolutionizing the AI industry with their superior capabilities.
Training these models requires large-scale GPU clusters and significant computing time, leading to frequent failures.
We introduce a novel reliability metric called emphTraining Overhead Ratio (TOR) to evaluate the reliability of fault-tolerant LLM training systems.
arXiv Detail & Related papers (2024-08-14T11:55:28Z) - Towards Certified Unlearning for Deep Neural Networks [50.816473152067104]
certified unlearning has been extensively studied in convex machine learning models.
We propose several techniques to bridge the gap between certified unlearning and deep neural networks (DNNs)
arXiv Detail & Related papers (2024-08-01T21:22:10Z) - Cross-Input Certified Training for Universal Perturbations [4.456428506059651]
Current certified training methods train models robust to single-input perturbations but achieve suboptimal clean and UAP accuracy.
We propose a novel method, CITRUS, for certified training of networks robust against UAP attackers.
We show in an extensive evaluation across different datasets, architectures, and perturbation magnitudes that our method outperforms traditional certified training methods on standard accuracy (up to 10.3%) and achieves SOTA performance on the more practical certified UAP accuracy metric.
arXiv Detail & Related papers (2024-05-15T08:33:41Z) - Optimistic Verifiable Training by Controlling Hardware Nondeterminism [22.85808027490485]
We propose a method that combines training in a higher precision than the target model, rounding after intermediate steps, and storing rounding decisions.
We achieve exact training replication at FP32 precision for both full-training and fine-tuning of ResNet-50 (23M) and GPT-2 (117M) models.
arXiv Detail & Related papers (2024-03-14T17:44:35Z) - Quantization-aware Interval Bound Propagation for Training Certifiably
Robust Quantized Neural Networks [58.195261590442406]
We study the problem of training and certifying adversarially robust quantized neural networks (QNNs)
Recent work has shown that floating-point neural networks that have been verified to be robust can become vulnerable to adversarial attacks after quantization.
We present quantization-aware interval bound propagation (QA-IBP), a novel method for training robust QNNs.
arXiv Detail & Related papers (2022-11-29T13:32:38Z) - Transferring Adversarial Robustness Through Robust Representation
Matching [3.5934248574481717]
Adrial training is one of the few known defenses able to reliably withstand such attacks against neural networks.
We propose Robust Representation Matching (RRM), a low-cost method to transfer the robustness of an adversarially trained model to a new model.
RRM is superior with respect to both model performance and adversarial training time.
arXiv Detail & Related papers (2022-02-21T05:15:40Z) - CRFL: Certifiably Robust Federated Learning against Backdoor Attacks [59.61565692464579]
This paper provides the first general framework, Certifiably Robust Federated Learning (CRFL), to train certifiably robust FL models against backdoors.
Our method exploits clipping and smoothing on model parameters to control the global model smoothness, which yields a sample-wise robustness certification on backdoors with limited magnitude.
arXiv Detail & Related papers (2021-06-15T16:50:54Z) - Fast Training of Provably Robust Neural Networks by SingleProp [71.19423596238568]
We develop a new regularizer that is both more efficient than existing certified defenses.
We demonstrate improvements in training speed and comparable certified accuracy compared to state-of-the-art certified defenses.
arXiv Detail & Related papers (2021-02-01T22:12:51Z) - Rethinking Clustering for Robustness [56.14672993686335]
ClusTR is a clustering-based and adversary-free training framework to learn robust models.
textitClusTR outperforms adversarially-trained networks by up to $4%$ under strong PGD attacks.
arXiv Detail & Related papers (2020-06-13T16:55:51Z) - HYDRA: Pruning Adversarially Robust Neural Networks [58.061681100058316]
Deep learning faces two key challenges: lack of robustness against adversarial attacks and large neural network size.
We propose to make pruning techniques aware of the robust training objective and let the training objective guide the search for which connections to prune.
We demonstrate that our approach, titled HYDRA, achieves compressed networks with state-of-the-art benign and robust accuracy, simultaneously.
arXiv Detail & Related papers (2020-02-24T19:54:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.