Sharpness-Aware Minimization Alone can Improve Adversarial Robustness
- URL: http://arxiv.org/abs/2305.05392v2
- Date: Sat, 1 Jul 2023 05:07:49 GMT
- Title: Sharpness-Aware Minimization Alone can Improve Adversarial Robustness
- Authors: Zeming Wei, Jingyu Zhu, Yihao Zhang
- Abstract summary: We explore Sharpness-Aware Minimization (SAM) in the context of adversarial robustness.
We find that using only SAM can achieve superior adversarial robustness without sacrificing clean accuracy compared to standard training.
We show that SAM and adversarial training (AT) differ in terms of perturbation strength, leading to different accuracy and robustness trade-offs.
- Score: 7.9810915020234035
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Sharpness-Aware Minimization (SAM) is an effective method for improving
generalization ability by regularizing loss sharpness. In this paper, we
explore SAM in the context of adversarial robustness. We find that using only
SAM can achieve superior adversarial robustness without sacrificing clean
accuracy compared to standard training, which is an unexpected benefit. We also
discuss the relation between SAM and adversarial training (AT), a popular
method for improving the adversarial robustness of DNNs. In particular, we show
that SAM and AT differ in terms of perturbation strength, leading to different
accuracy and robustness trade-offs. We provide theoretical evidence for these
claims in a simplified model. Finally, while AT suffers from decreased clean
accuracy and computational overhead, we suggest that SAM can be regarded as a
lightweight substitute for AT under certain requirements. Code is available at
https://github.com/weizeming/SAM_AT.
Related papers
- Improving SAM Requires Rethinking its Optimization Formulation [57.601718870423454]
Sharpness-Aware Minimization (SAM) is originally formulated as a zero-sum game where the weights of a network and a bounded perturbation try to minimize/maximize, respectively, the same differentiable loss.
We argue that SAM should instead be reformulated using the 0-1 loss. As a continuous relaxation, we follow the simple conventional approach where the minimizing (maximizing) player uses an upper bound (lower bound) surrogate to the 0-1 loss. This leads to a novel formulation of SAM as a bilevel optimization problem, dubbed as BiSAM.
arXiv Detail & Related papers (2024-07-17T20:22:33Z) - On the Duality Between Sharpness-Aware Minimization and Adversarial Training [14.863336218063646]
Adversarial Training (AT) is one of the most effective defenses against adversarial attacks, yet suffers from inevitably decreased clean accuracy.
Instead of perturbing the samples, Sharpness-Aware Minimization (SAM) perturbs the model weights during training to find a more flat loss landscape.
We find that using SAM alone can improve adversarial robustness.
arXiv Detail & Related papers (2024-02-23T07:22:55Z) - Stabilizing Sharpness-aware Minimization Through A Simple
Renormalization Strategy [12.927965934262847]
Training neural networks with sharpness-aware (SAM) can be highly unstable.
We propose a simple renormalization strategy, dubbed StableSAM, so that the norm of the surrogate gradient maintains the same as that of the exact gradient.
We show how StableSAM extends this regime of learning rate and when it can consistently perform better than SAM with minor modification.
arXiv Detail & Related papers (2024-01-14T10:53:36Z) - Stable Segment Anything Model [79.9005670886038]
The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts.
This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities.
Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
arXiv Detail & Related papers (2023-11-27T12:51:42Z) - Enhancing Sharpness-Aware Optimization Through Variance Suppression [48.908966673827734]
This work embraces the geometry of the loss function, where neighborhoods of 'flat minima' heighten generalization ability.
It seeks 'flat valleys' by minimizing the maximum loss caused by an adversary perturbing parameters within the neighborhood.
Although critical to account for sharpness of the loss function, such an 'over-friendly adversary' can curtail the outmost level of generalization.
arXiv Detail & Related papers (2023-09-27T13:18:23Z) - Systematic Investigation of Sparse Perturbed Sharpness-Aware
Minimization Optimizer [158.2634766682187]
Deep neural networks often suffer from poor generalization due to complex and non- unstructured loss landscapes.
SharpnessAware Minimization (SAM) is a popular solution that smooths the loss by minimizing the change of landscape when adding a perturbation.
In this paper, we propose Sparse SAM (SSAM), an efficient and effective training scheme that achieves perturbation by a binary mask.
arXiv Detail & Related papers (2023-06-30T09:33:41Z) - Sharpness-Aware Minimization Revisited: Weighted Sharpness as a
Regularization Term [4.719514928428503]
We propose a more general method, called WSAM, by incorporating sharpness as a regularization term.
We prove its generalization bound through the combination of PAC and Bayes-PAC techniques.
The results demonstrate that WSAM achieves improved generalization, or is at least highly competitive, compared to the vanilla, SAM and its variants.
arXiv Detail & Related papers (2023-05-25T08:00:34Z) - Improved Deep Neural Network Generalization Using m-Sharpness-Aware
Minimization [14.40189851070842]
Sharpness-Aware Minimization (SAM) modifies the underlying loss function to guide descent methods towards flatter minima.
Recent work suggests that mSAM can outperform SAM in terms of test accuracy.
This paper presents a comprehensive empirical evaluation of mSAM on various tasks and datasets.
arXiv Detail & Related papers (2022-12-07T00:37:55Z) - Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation
Approach [132.37966970098645]
One of the popular solutions is Sharpness-Aware Minimization (SAM), which minimizes the change of weight loss when adding a perturbation.
In this paper, we propose an efficient effective training scheme coined as Sparse SAM (SSAM), which achieves double overhead of common perturbations.
In addition, we theoretically prove that S can converge at the same SAM, i.e., $O(log T/sqrtTTTTTTTTTTTTTTTTT
arXiv Detail & Related papers (2022-10-11T06:30:10Z) - Efficient Sharpness-aware Minimization for Improved Training of Neural
Networks [146.2011175973769]
This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance.
M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection.
We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
arXiv Detail & Related papers (2021-10-07T02:20:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.