Exploring the Relationship between Architecture and Adversarially Robust Generalization

• URL: http://arxiv.org/abs/2209.14105v1
• Date: Wed, 28 Sep 2022 13:55:28 GMT
• Title: Exploring the Relationship between Architecture and Adversarially Robust Generalization
• Authors: Shiyu Tang, Siyuan Liang, Ruihao Gong, Aishan Liu, Xianglong Liu, Dacheng Tao
• Abstract summary: Adversarial training is one of the most effective remedies for defending adversarial examples. It often suffers from the huge robustness generalization gap on unseen testing adversaries. This paper tries to bridge the gap by systematically examining the most representative architectures.
• Score: 110.00366382964472
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adversarial training has been demonstrated to be one of the most effective remedies for defending adversarial examples, yet it often suffers from the huge robustness generalization gap on unseen testing adversaries, deemed as the \emph{adversarially robust generalization problem}. Despite the preliminary understandings devoted on adversarially robust generalization, little is known from the architectural perspective. Thus, this paper tries to bridge the gap by systematically examining the most representative architectures (e.g., Vision Transformers and CNNs). In particular, we first comprehensively evaluated \emph{20} adversarially trained architectures on ImageNette and CIFAR-10 datasets towards several adversaries (multiple $\ell_p$-norm adversarial attacks), and found that Vision Transformers (e.g., PVT, CoAtNet) often yield better adversarially robust generalization. To further understand what architectural ingredients favor adversarially robust generalization, we delve into several key building blocks and revealed the fact via the lens of Rademacher complexity that the higher weight sparsity contributes significantly towards the better adversarially robust generalization of Vision Transformers, which can be often achieved by attention layers. Our extensive studies discovered the close relationship between architectural design and adversarially robust generalization, and instantiated several important insights. We hope our findings could help to better understand the mechanism towards designing robust deep learning architectures.

Related papers

• Towards Accurate and Robust Architectures via Neural Architecture Search [3.4014222238829497]
  adversarial training improves accuracy and robustness by adjusting the weight connection affiliated to the architecture. We propose ARNAS to search for accurate and robust architectures for adversarial training.
  arXiv  Detail & Related papers  (2024-05-09T02:16:50Z)
• Understanding Robustness of Visual State Space Models for Image Classification [19.629800707546543]
  Visual State Space Model (VMamba) has emerged as a promising architecture, exhibiting remarkable performance in various computer vision tasks. We investigate its robustness to adversarial attacks, employing both whole-image and patch-specific adversarial attacks. We explore VMamba's gradients and back-propagation during white-box attacks, uncovering unique vulnerabilities and defensive capabilities.
  arXiv  Detail & Related papers  (2024-03-16T14:23:17Z)
• A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking [54.89987482509155]
  robustness of deep neural networks is usually lacking under adversarial examples, common corruptions, and distribution shifts. We establish a comprehensive benchmark robustness called textbfARES-Bench on the image classification task. By designing the training settings accordingly, we achieve the new state-of-the-art adversarial robustness.
  arXiv  Detail & Related papers  (2023-02-28T04:26:20Z)
• On the interplay of adversarial robustness and architecture components: patches, convolution and attention [65.20660287833537]
  We study the effect of adversarial training on the interpretability of the learnt features and robustness to unseen threat models. An ablation from ResNet to ConvNeXt reveals key architectural changes leading to almost $10%$ higher $ell_infty$-robustness.
  arXiv  Detail & Related papers  (2022-09-14T22:02:32Z)
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks [98.21130211336964]
  Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. In this paper, we investigate the impact of network width and depth on the robustness of adversarially trained DNNs.
  arXiv  Detail & Related papers  (2021-10-07T23:13:33Z)
• Generalization by design: Shortcuts to Generalization in Deep Learning [7.751691910877239]
  We show that good generalization may be instigated by bounded spectral products over layers leading to a novel geometric regularizer. Backed up by theory we further demonstrate that "generalization by design" is practically possible and that good generalization may be encoded into the structure of the network.
  arXiv  Detail & Related papers  (2021-07-05T20:01:23Z)
• In Search of Robust Measures of Generalization [79.75709926309703]
  We develop bounds on generalization error, optimization error, and excess risk. When evaluated empirically, most of these bounds are numerically vacuous. We argue that generalization measures should instead be evaluated within the framework of distributional robustness.
  arXiv  Detail & Related papers  (2020-10-22T17:54:25Z)
• Optimism in the Face of Adversity: Understanding and Improving Deep Learning through Adversarial Robustness [63.627760598441796]
  We provide an in-depth review of the field of adversarial robustness in deep learning. We highlight the intuitive connection between adversarial examples and the geometry of deep neural networks. We provide an overview of the main emerging applications of adversarial robustness beyond security.
  arXiv  Detail & Related papers  (2020-10-19T16:03:46Z)
• Architectural Resilience to Foreground-and-Background Adversarial Noise [0.0]
  Adrial attacks in the form of imperceptible perturbations of normal images have been extensively studied. We propose distinct model-agnostic benchmark perturbations of images to investigate resilience and robustness of different network architectures.
  arXiv  Detail & Related papers  (2020-03-23T01:38:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.