Ensemble-in-One: Learning Ensemble within Random Gated Networks for
Enhanced Adversarial Robustness
- URL: http://arxiv.org/abs/2103.14795v1
- Date: Sat, 27 Mar 2021 03:13:03 GMT
- Title: Ensemble-in-One: Learning Ensemble within Random Gated Networks for
Enhanced Adversarial Robustness
- Authors: Yi Cai, Xuefei Ning, Huazhong Yang, Yu Wang
- Abstract summary: Adversarial attacks have rendered high security risks on modern deep learning systems.
We propose ensemble-in-one (EIO) to train an ensemble within one random gated network (RGN)
EIO consistently outperforms previous ensemble training methods with even less computational overhead.
- Score: 18.514706498043214
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Adversarial attacks have rendered high security risks on modern deep learning
systems. Adversarial training can significantly enhance the robustness of
neural network models by suppressing the non-robust features. However, the
models often suffer from significant accuracy loss on clean data. Ensemble
training methods have emerged as promising solutions for defending against
adversarial attacks by diversifying the vulnerabilities among the sub-models,
simultaneously maintaining comparable accuracy as standard training. However,
existing ensemble methods are with poor scalability, owing to the rapid
complexity increase when including more sub-models in the ensemble. Moreover,
in real-world applications, it is difficult to deploy an ensemble with multiple
sub-models, owing to the tight hardware resource budget and latency
requirement. In this work, we propose ensemble-in-one (EIO), a simple but
efficient way to train an ensemble within one random gated network (RGN). EIO
augments the original model by replacing the parameterized layers with
multi-path random gated blocks (RGBs) to construct a RGN. By diversifying the
vulnerability of the numerous paths within the RGN, better robustness can be
achieved. It provides high scalability because the paths within an EIO network
exponentially increase with the network depth. Our experiments demonstrate that
EIO consistently outperforms previous ensemble training methods with even less
computational overhead.
Related papers
- LoRA-Ensemble: Efficient Uncertainty Modelling for Self-attention Networks [52.46420522934253]
We introduce LoRA-Ensemble, a parameter-efficient deep ensemble method for self-attention networks.
By employing a single pre-trained self-attention network with weights shared across all members, we train member-specific low-rank matrices for the attention projections.
Our method exhibits superior calibration compared to explicit ensembles and achieves similar or better accuracy across various prediction tasks and datasets.
arXiv Detail & Related papers (2024-05-23T11:10:32Z) - Boosting Adversarial Training via Fisher-Rao Norm-based Regularization [9.975998980413301]
We propose a novel regularization framework, called Logit-Oriented Adversarial Training (LOAT), which can mitigate the trade-off between robustness and accuracy.
Our experiments demonstrate that the proposed regularization strategy can boost the performance of the prevalent adversarial training algorithms.
arXiv Detail & Related papers (2024-03-26T09:22:37Z) - Learning Robust Kernel Ensembles with Kernel Average Pooling [3.6540368812166872]
We introduce Kernel Average Pooling (KAP), a neural network building block that applies the mean filter along the kernel dimension of the layer activation tensor.
We show that ensembles of kernels with similar functionality naturally emerge in convolutional neural networks equipped with KAP and trained with backpropagation.
arXiv Detail & Related papers (2022-09-30T19:49:14Z) - Adversarial Vulnerability of Randomized Ensembles [12.082239973914326]
We show that randomized ensembles are more vulnerable to imperceptible adversarial perturbations than even standard AT models.
We propose a theoretically-sound and efficient adversarial attack algorithm (ARC) capable of compromising random ensembles even in cases where adaptive PGD fails to do so.
arXiv Detail & Related papers (2022-06-14T10:37:58Z) - Distributed Adversarial Training to Robustify Deep Neural Networks at
Scale [100.19539096465101]
Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification.
To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training.
We propose a large-batch adversarial training framework implemented over multiple machines.
arXiv Detail & Related papers (2022-06-13T15:39:43Z) - Sparsity Winning Twice: Better Robust Generalization from More Efficient
Training [94.92954973680914]
We introduce two alternatives for sparse adversarial training: (i) static sparsity and (ii) dynamic sparsity.
We find both methods to yield win-win: substantially shrinking the robust generalization gap and alleviating the robust overfitting.
Our approaches can be combined with existing regularizers, establishing new state-of-the-art results in adversarial training.
arXiv Detail & Related papers (2022-02-20T15:52:08Z) - Interpolated Joint Space Adversarial Training for Robust and
Generalizable Defenses [82.3052187788609]
Adversarial training (AT) is considered to be one of the most reliable defenses against adversarial attacks.
Recent works show generalization improvement with adversarial samples under novel threat models.
We propose a novel threat model called Joint Space Threat Model (JSTM)
Under JSTM, we develop novel adversarial attacks and defenses.
arXiv Detail & Related papers (2021-12-12T21:08:14Z) - GOAT: GPU Outsourcing of Deep Learning Training With Asynchronous
Probabilistic Integrity Verification Inside Trusted Execution Environment [0.0]
Machine learning models based on Deep Neural Networks (DNNs) are increasingly deployed in a range of applications ranging from self-driving cars to COVID-19 treatment discovery.
To support the computational power necessary to learn a DNN, cloud environments with dedicated hardware support have emerged as critical infrastructure.
Various approaches have been developed to address these challenges, building on trusted execution environments (TEE)
arXiv Detail & Related papers (2020-10-17T20:09:05Z) - R-FORCE: Robust Learning for Random Recurrent Neural Networks [6.285241353736006]
We propose a robust training method to enhance robustness of RRNN.
FORCE learning approach was shown to be applicable even for the challenging task of target-learning.
Our experiments indicate that R-FORCE facilitates significantly more stable and accurate target-learning for a wide class of RRNN.
arXiv Detail & Related papers (2020-03-25T22:08:03Z) - HYDRA: Pruning Adversarially Robust Neural Networks [58.061681100058316]
Deep learning faces two key challenges: lack of robustness against adversarial attacks and large neural network size.
We propose to make pruning techniques aware of the robust training objective and let the training objective guide the search for which connections to prune.
We demonstrate that our approach, titled HYDRA, achieves compressed networks with state-of-the-art benign and robust accuracy, simultaneously.
arXiv Detail & Related papers (2020-02-24T19:54:53Z) - An Image Enhancing Pattern-based Sparsity for Real-time Inference on
Mobile Devices [58.62801151916888]
We introduce a new sparsity dimension, namely pattern-based sparsity that comprises pattern and connectivity sparsity, and becoming both highly accurate and hardware friendly.
Our approach on the new pattern-based sparsity naturally fits into compiler optimization for highly efficient DNN execution on mobile platforms.
arXiv Detail & Related papers (2020-01-20T16:17:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.