Enhancing Adversarial Training with Second-Order Statistics of Weights
- URL: http://arxiv.org/abs/2203.06020v1
- Date: Fri, 11 Mar 2022 15:40:57 GMT
- Title: Enhancing Adversarial Training with Second-Order Statistics of Weights
- Authors: Gaojie Jin, Xinping Yi, Wei Huang, Sven Schewe, Xiaowei Huang
- Abstract summary: We show that treating model weights as random variables allows for enhancing adversarial training through textbfSecond-Order textbfStatistics textbfOptimization.
We conduct an extensive set of experiments, which show that S$2$O not only improves the robustness and generalization of the trained neural networks when used in isolation, but also integrates easily in state-of-the-art adversarial training techniques.
- Score: 23.90998469971413
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Adversarial training has been shown to be one of the most effective
approaches to improve the robustness of deep neural networks. It is formalized
as a min-max optimization over model weights and adversarial perturbations,
where the weights can be optimized through gradient descent methods like SGD.
In this paper, we show that treating model weights as random variables allows
for enhancing adversarial training through \textbf{S}econd-Order
\textbf{S}tatistics \textbf{O}ptimization (S$^2$O) with respect to the weights.
By relaxing a common (but unrealistic) assumption of previous PAC-Bayesian
frameworks that all weights are statistically independent, we derive an
improved PAC-Bayesian adversarial generalization bound, which suggests that
optimizing second-order statistics of weights can effectively tighten the
bound. In addition to this theoretical insight, we conduct an extensive set of
experiments, which show that S$^2$O not only improves the robustness and
generalization of the trained neural networks when used in isolation, but also
integrates easily in state-of-the-art adversarial training techniques like
TRADES, AWP, MART, and AVMixup, leading to a measurable improvement of these
techniques. The code is available at \url{https://github.com/Alexkael/S2O}.
Related papers
- Improving Generalization of Adversarial Training via Robust Critical
Fine-Tuning [19.91117174405902]
Deep neural networks are susceptible to adversarial examples, posing a significant security risk in critical applications.
This paper proposes Robustness Critical FineTuning (RiFT), a novel approach to enhance generalization without compromising adversarial robustness.
arXiv Detail & Related papers (2023-08-01T09:02:34Z) - Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
We propose a novel doubly-robust instance reweighted adversarial framework.
Our importance weights are obtained by optimizing the KL-divergence regularized loss function.
Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
arXiv Detail & Related papers (2023-08-01T06:16:18Z) - TWINS: A Fine-Tuning Framework for Improved Transferability of
Adversarial Robustness and Generalization [89.54947228958494]
This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks.
We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework.
TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
arXiv Detail & Related papers (2023-03-20T14:12:55Z) - Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality [65.67315418971688]
Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR) are proposed.
Experiments on visual recognition demonstrate that our methods can simultaneously improve the covariance conditioning and generalization.
arXiv Detail & Related papers (2022-07-05T15:39:29Z) - Distributed Adversarial Training to Robustify Deep Neural Networks at
Scale [100.19539096465101]
Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification.
To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training.
We propose a large-batch adversarial training framework implemented over multiple machines.
arXiv Detail & Related papers (2022-06-13T15:39:43Z) - Self-Ensemble Adversarial Training for Improved Robustness [14.244311026737666]
Adversarial training is the strongest strategy against various adversarial attacks among all sorts of defense methods.
Recent works mainly focus on developing new loss functions or regularizers, attempting to find the unique optimal point in the weight space.
We devise a simple but powerful emphSelf-Ensemble Adversarial Training (SEAT) method for yielding a robust classifier by averaging weights of history models.
arXiv Detail & Related papers (2022-03-18T01:12:18Z) - Self-Progressing Robust Training [146.8337017922058]
Current robust training methods such as adversarial training explicitly uses an "attack" to generate adversarial examples.
We propose a new framework called SPROUT, self-progressing robust training.
Our results shed new light on scalable, effective and attack-independent robust training methods.
arXiv Detail & Related papers (2020-12-22T00:45:24Z) - Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness.
We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
arXiv Detail & Related papers (2020-10-26T04:44:43Z) - Training Sparse Neural Networks using Compressed Sensing [13.84396596420605]
We develop and test a novel method based on compressed sensing which combines the pruning and training into a single step.
Specifically, we utilize an adaptively weighted $ell1$ penalty on the weights during training, which we combine with a generalization of the regularized dual averaging (RDA) algorithm in order to train sparse neural networks.
arXiv Detail & Related papers (2020-08-21T19:35:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.