On the Duality Between Sharpness-Aware Minimization and Adversarial Training

• URL: http://arxiv.org/abs/2402.15152v2
• Date: Wed, 5 Jun 2024 08:39:57 GMT
• Title: On the Duality Between Sharpness-Aware Minimization and Adversarial Training
• Authors: Yihao Zhang, Hangzhou He, Jingyu Zhu, Huanran Chen, Yifei Wang, Zeming Wei,
• Abstract summary: Adversarial Training (AT) is one of the most effective defenses against adversarial attacks, yet suffers from inevitably decreased clean accuracy. Instead of perturbing the samples, Sharpness-Aware Minimization (SAM) perturbs the model weights during training to find a more flat loss landscape. We find that using SAM alone can improve adversarial robustness.
• Score: 14.863336218063646
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adversarial Training (AT), which adversarially perturb the input samples during training, has been acknowledged as one of the most effective defenses against adversarial attacks, yet suffers from inevitably decreased clean accuracy. Instead of perturbing the samples, Sharpness-Aware Minimization (SAM) perturbs the model weights during training to find a more flat loss landscape and improve generalization. However, as SAM is designed for better clean accuracy, its effectiveness in enhancing adversarial robustness remains unexplored. In this work, considering the duality between SAM and AT, we investigate the adversarial robustness derived from SAM. Intriguingly, we find that using SAM alone can improve adversarial robustness. To understand this unexpected property of SAM, we first provide empirical and theoretical insights into how SAM can implicitly learn more robust features, and conduct comprehensive experiments to show that SAM can improve adversarial robustness notably without sacrificing any clean accuracy, shedding light on the potential of SAM to be a substitute for AT when accuracy comes at a higher priority. Code is available at https://github.com/weizeming/SAM_AT.

Related papers

• Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training [47.25594539120258]
  We find that Sharpness-Aware Minimization (SAM) efficiently selects flatter minima late in training. Even a few epochs of SAM applied at the end of training yield nearly the same generalization and solution sharpness as full SAM training. We conjecture that the optimization method chosen in the late phase is more crucial in shaping the final solution's properties.
  arXiv  Detail & Related papers  (2024-10-14T10:56:42Z)
• Bilateral Sharpness-Aware Minimization for Flatter Minima [61.17349662062522]
  Sharpness-Aware Minimization (SAM) enhances generalization by reducing a Max-Sharpness (MaxS) In this paper, we propose to utilize the difference between the training loss and the minimum loss over the neighborhood surrounding the current weight, which we denote as Min-Sharpness (MinS) By merging MaxS and MinS, we created a better FI that indicates a flatter direction during the optimization. Specially, we combine this FI with SAM into the proposed Bilateral SAM (BSAM) which finds a more flatter minimum than that of SAM.
  arXiv  Detail & Related papers  (2024-09-20T03:01:13Z)
• Friendly Sharpness-Aware Minimization [62.57515991835801]
  Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. We investigate the key role of batch-specific gradient noise within the adversarial perturbation, i.e., the current minibatch gradient. By decomposing the adversarial gradient noise components, we discover that relying solely on the full gradient degrades generalization while excluding it leads to improved performance.
  arXiv  Detail & Related papers  (2024-03-19T01:39:33Z)
• Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy [12.050160495730381]
  sharpness-aware generalization (SAM) has attracted much attention because of its surprising effectiveness in improving performance. We propose a simple renormalization strategy, dubbed Stable SAM (SSAM), so that the gradient norm of the descent step maintains the same as that of the ascent step. Our strategy is easy to implement and flexible enough to integrate with SAM and its variants, almost at no computational cost.
  arXiv  Detail & Related papers  (2024-01-14T10:53:36Z)
• Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer [158.2634766682187]
  Deep neural networks often suffer from poor generalization due to complex and non- unstructured loss landscapes. SharpnessAware Minimization (SAM) is a popular solution that smooths the loss by minimizing the change of landscape when adding a perturbation. In this paper, we propose Sparse SAM (SSAM), an efficient and effective training scheme that achieves perturbation by a binary mask.
  arXiv  Detail & Related papers  (2023-06-30T09:33:41Z)
• On the Robustness of Segment Anything [46.669794757467166]
  We aim to study the testing-time robustness of SAM under adversarial scenarios and common corruptions. We find that SAM exhibits remarkable robustness against various corruptions, except for blur-related corruption.
  arXiv  Detail & Related papers  (2023-05-25T16:28:30Z)
• Sharpness-Aware Minimization Alone can Improve Adversarial Robustness [7.9810915020234035]
  We explore Sharpness-Aware Minimization (SAM) in the context of adversarial robustness. We find that using only SAM can achieve superior adversarial robustness without sacrificing clean accuracy compared to standard training. We show that SAM and adversarial training (AT) differ in terms of perturbation strength, leading to different accuracy and robustness trade-offs.
  arXiv  Detail & Related papers  (2023-05-09T12:39:21Z)
• Improved Deep Neural Network Generalization Using m-Sharpness-Aware Minimization [14.40189851070842]
  Sharpness-Aware Minimization (SAM) modifies the underlying loss function to guide descent methods towards flatter minima. Recent work suggests that mSAM can outperform SAM in terms of test accuracy. This paper presents a comprehensive empirical evaluation of mSAM on various tasks and datasets.
  arXiv  Detail & Related papers  (2022-12-07T00:37:55Z)
• Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach [132.37966970098645]
  One of the popular solutions is Sharpness-Aware Minimization (SAM), which minimizes the change of weight loss when adding a perturbation. In this paper, we propose an efficient effective training scheme coined as Sparse SAM (SSAM), which achieves double overhead of common perturbations. In addition, we theoretically prove that S can converge at the same SAM, i.e., $O(log T/sqrtTTTTTTTTTTTTTTTTT
  arXiv  Detail & Related papers  (2022-10-11T06:30:10Z)
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks [146.2011175973769]
  This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance. M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
  arXiv  Detail & Related papers  (2021-10-07T02:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.