SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
- URL: http://arxiv.org/abs/2003.09347v3
- Date: Mon, 8 Nov 2021 10:53:28 GMT
- Title: SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
- Authors: Chawin Sitawarin, Supriyo Chakraborty, David Wagner
- Abstract summary: We find that curriculum learning, a scheme that emphasizes on starting "easy" and gradually ramping up on the "difficulty" of training, smooths the adversarial loss landscape for a suitably chosen difficulty metric.
We demonstrate that SAT stabilizes network training even for a large perturbation norm and allows the network to operate at a better clean accuracy versus trade-off curve compared to AT.
- Score: 11.406879470613186
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Adversarial training (AT) has become a popular choice for training robust
networks. However, it tends to sacrifice clean accuracy heavily in favor of
robustness and suffers from a large generalization error. To address these
concerns, we propose Smooth Adversarial Training (SAT), guided by our analysis
on the eigenspectrum of the loss Hessian. We find that curriculum learning, a
scheme that emphasizes on starting "easy" and gradually ramping up on the
"difficulty" of training, smooths the adversarial loss landscape for a suitably
chosen difficulty metric. We present a general formulation for curriculum
learning in the adversarial setting and propose two difficulty metrics based on
the maximal Hessian eigenvalue (H-SAT) and the softmax probability (P-SA). We
demonstrate that SAT stabilizes network training even for a large perturbation
norm and allows the network to operate at a better clean accuracy versus
robustness trade-off curve compared to AT. This leads to a significant
improvement in both clean accuracy and robustness compared to AT, TRADES, and
other baselines. To highlight a few results, our best model improves normal and
robust accuracy by 6% and 1% on CIFAR-100 compared to AT, respectively. On
Imagenette, a ten-class subset of ImageNet, our model outperforms AT by 23% and
3% on normal and robust accuracy respectively.
Related papers
- Omnipotent Adversarial Training in the Wild [20.239704959690936]
We propose Omniversapotent Adrial Training (OAT) strategy to train a model on an imbalanced and noisy dataset.
OAT consists of two innovative methodologies to address the imperfection in the training set.
OAT outperforms other baselines by more than 20% clean accuracy improvement and 10% robust accuracy improvement.
arXiv Detail & Related papers (2023-07-14T07:09:57Z) - Enhancing Adversarial Training via Reweighting Optimization Trajectory [72.75558017802788]
A number of approaches have been proposed to address drawbacks such as extra regularization, adversarial weights, and training with more data.
We propose a new method named textbfWeighted Optimization Trajectories (WOT) that leverages the optimization trajectories of adversarial training in time.
Our results show that WOT integrates seamlessly with the existing adversarial training methods and consistently overcomes the robust overfitting issue.
arXiv Detail & Related papers (2023-06-25T15:53:31Z) - RUSH: Robust Contrastive Learning via Randomized Smoothing [31.717748554905015]
In this paper, we show a surprising fact that contrastive pre-training has an interesting yet implicit connection with robustness.
We design a powerful robust algorithm against adversarial attacks, RUSH, that combines the standard contrastive pre-training and randomized smoothing.
Our work has an improvement of over 15% in robust accuracy and a slight improvement in standard accuracy, compared to the state-of-the-arts.
arXiv Detail & Related papers (2022-07-11T18:45:14Z) - Removing Batch Normalization Boosts Adversarial Training [83.08844497295148]
Adversarial training (AT) defends deep neural networks against adversarial attacks.
A major bottleneck is the widely used batch normalization (BN), which struggles to model the different statistics of clean and adversarial training samples in AT.
Our normalizer-free robust training (NoFrost) method extends recent advances in normalizer-free networks to AT.
arXiv Detail & Related papers (2022-07-04T01:39:37Z) - A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via
Adversarial Fine-tuning [90.44219200633286]
We propose a simple yet very effective adversarial fine-tuning approach based on a $textitslow start, fast decay$ learning rate scheduling strategy.
Experimental results show that the proposed adversarial fine-tuning approach outperforms the state-of-the-art methods on CIFAR-10, CIFAR-100 and ImageNet datasets.
arXiv Detail & Related papers (2020-12-25T20:50:15Z) - Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness
and Accuracy for Free [115.81899803240758]
Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy.
This paper asks how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies.
Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework.
arXiv Detail & Related papers (2020-10-22T16:06:34Z) - Smooth Adversarial Training [120.44430400607483]
It is commonly believed that networks cannot be both accurate and robust.
Here we present evidence to challenge these common beliefs by a careful study about adversarial training.
We propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training.
arXiv Detail & Related papers (2020-06-25T16:34:39Z) - Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning [134.15174177472807]
We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time.
We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins.
arXiv Detail & Related papers (2020-03-28T18:28:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.