Smooth Adversarial Training

• URL: http://arxiv.org/abs/2006.14536v2
• Date: Sun, 11 Jul 2021 00:56:58 GMT
• Title: Smooth Adversarial Training
• Authors: Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, Quoc V. Le
• Abstract summary: It is commonly believed that networks cannot be both accurate and robust. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. We propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training.
• Score: 120.44430400607483
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: It is commonly believed that networks cannot be both accurate and robust, that gaining robustness means losing accuracy. It is also generally believed that, unless making networks larger, network architectural elements would otherwise matter little in improving adversarial robustness. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. Our key observation is that the widely-used ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Hence we propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training. The purpose of smooth activation functions in SAT is to allow it to find harder adversarial examples and compute better gradient updates during adversarial training. Compared to standard adversarial training, SAT improves adversarial robustness for "free", i.e., no drop in accuracy and no increase in computational cost. For example, without introducing additional computations, SAT significantly enhances ResNet-50's robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https://github.com/cihangxie/SmoothAdversarialTraining.

Related papers

• MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsification [32.70084821901212]
  MeanSparse is a method to improve the robustness of Convolutional and attention-based Neural Networks against adversarial examples. Our experiments show that MeanSparse achieves a new robustness record of 75.28%.
  arXiv  Detail & Related papers  (2024-06-09T22:14:55Z)
• Enhancing Adversarial Training via Reweighting Optimization Trajectory [72.75558017802788]
  A number of approaches have been proposed to address drawbacks such as extra regularization, adversarial weights, and training with more data. We propose a new method named textbfWeighted Optimization Trajectories (WOT) that leverages the optimization trajectories of adversarial training in time. Our results show that WOT integrates seamlessly with the existing adversarial training methods and consistently overcomes the robust overfitting issue.
  arXiv  Detail & Related papers  (2023-06-25T15:53:31Z)
• RUSH: Robust Contrastive Learning via Randomized Smoothing [31.717748554905015]
  In this paper, we show a surprising fact that contrastive pre-training has an interesting yet implicit connection with robustness. We design a powerful robust algorithm against adversarial attacks, RUSH, that combines the standard contrastive pre-training and randomized smoothing. Our work has an improvement of over 15% in robust accuracy and a slight improvement in standard accuracy, compared to the state-of-the-arts.
  arXiv  Detail & Related papers  (2022-07-11T18:45:14Z)
• AdvRush: Searching for Adversarially Robust Neural Architectures [17.86463546971522]
  We propose AdvRush, a novel adversarial robustness-aware neural architecture search algorithm. Through a regularizer that favors a candidate architecture with a smoother input loss landscape, AdvRush successfully discovers an adversarially robust neural architecture.
  arXiv  Detail & Related papers  (2021-08-03T04:27:33Z)
• To be Robust or to be Fair: Towards Fairness in Adversarial Training [83.42241071662897]
  We find that adversarial training algorithms tend to introduce severe disparity of accuracy and robustness between different groups of data. We propose a Fair-Robust-Learning (FRL) framework to mitigate this unfairness problem when doing adversarial defenses.
  arXiv  Detail & Related papers  (2020-10-13T02:21:54Z)
• Exploring Model Robustness with Adaptive Networks and Improved Adversarial Training [56.82000424924979]
  We propose a conditional normalization module to adapt networks when conditioned on input samples. Our adaptive networks, once adversarially trained, can outperform their non-adaptive counterparts on both clean validation accuracy and robustness.
  arXiv  Detail & Related papers  (2020-05-30T23:23:56Z)
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning [134.15174177472807]
  We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins.
  arXiv  Detail & Related papers  (2020-03-28T18:28:33Z)
• SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing [11.406879470613186]
  We find that curriculum learning, a scheme that emphasizes on starting "easy" and gradually ramping up on the "difficulty" of training, smooths the adversarial loss landscape for a suitably chosen difficulty metric. We demonstrate that SAT stabilizes network training even for a large perturbation norm and allows the network to operate at a better clean accuracy versus trade-off curve compared to AT.
  arXiv  Detail & Related papers  (2020-03-18T20:59:45Z)
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference [119.19779637025444]
  Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) This paper studies multi-exit networks associated with input-adaptive inference, showing their strong promise in achieving a "sweet point" in cooptimizing model accuracy, robustness and efficiency.
  arXiv  Detail & Related papers  (2020-02-24T00:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.