Revisiting Residual Networks for Adversarial Robustness: An Architectural Perspective

• URL: http://arxiv.org/abs/2212.11005v1
• Date: Wed, 21 Dec 2022 13:19:25 GMT
• Title: Revisiting Residual Networks for Adversarial Robustness: An Architectural Perspective
• Authors: Shihua Huang, Zhichao Lu, Kalyanmoy Deb, Vishnu Naresh Boddeti
• Abstract summary: We focus on residual networks and consider architecture design at the block level, i.e., topology, kernel size, activation, and normalization. We present a portfolio of adversarially robust residual networks, RobustResNets, spanning a broad spectrum of model capacities.
• Score: 22.59262601575886
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Efforts to improve the adversarial robustness of convolutional neural networks have primarily focused on developing more effective adversarial training methods. In contrast, little attention was devoted to analyzing the role of architectural elements (such as topology, depth, and width) on adversarial robustness. This paper seeks to bridge this gap and present a holistic study on the impact of architectural design on adversarial robustness. We focus on residual networks and consider architecture design at the block level, i.e., topology, kernel size, activation, and normalization, as well as at the network scaling level, i.e., depth and width of each block in the network. In both cases, we first derive insights through systematic ablative experiments. Then we design a robust residual block, dubbed RobustResBlock, and a compound scaling rule, dubbed RobustScaling, to distribute depth and width at the desired FLOP count. Finally, we combine RobustResBlock and RobustScaling and present a portfolio of adversarially robust residual networks, RobustResNets, spanning a broad spectrum of model capacities. Experimental validation across multiple datasets and adversarial attacks demonstrate that RobustResNets consistently outperform both the standard WRNs and other existing robust architectures, achieving state-of-the-art AutoAttack robust accuracy of 61.1% without additional data and 63.7% with 500K external data while being $2\times$ more compact in terms of parameters. Code is available at \url{ https://github.com/zhichao-lu/robust-residual-network}

Related papers

• (PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork [60.889175951038496]
  Large-scale neural networks have demonstrated remarkable performance in different domains like vision and language processing. One of the key questions of structural pruning is how to estimate the channel significance. We propose a novel algorithmic framework, namely textttPASS. It is a tailored hyper-network to take both visual prompts and network weight statistics as input, and output layer-wise channel sparsity in a recurrent manner.
  arXiv  Detail & Related papers  (2024-07-24T16:47:45Z)
• Neural Architecture Design and Robustness: A Dataset [11.83842808044211]
  We introduce a database on neural architecture design and robustness evaluations. We evaluate all these networks on a range of common adversarial attacks and corruption types. We find that carefully crafting the topology of a network can have substantial impact on its robustness.
  arXiv  Detail & Related papers  (2023-06-11T16:02:14Z)
• RobArch: Designing Robust Architectures against Adversarial Attacks [7.7720465119590845]
  Adrial Training is the most effective approach for improving the robustness of Deep Neural Networks (DNNs) We present the first large-scale systematic study on the robustness of DNN architecture components under fixed parameter budgets. We demonstrate 18 actionable robust network design guidelines that empower model developers to gain deep insights.
  arXiv  Detail & Related papers  (2023-01-08T21:19:52Z)
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks [98.21130211336964]
  Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. In this paper, we investigate the impact of network width and depth on the robustness of adversarially trained DNNs.
  arXiv  Detail & Related papers  (2021-10-07T23:13:33Z)
• Tiny Adversarial Mulit-Objective Oneshot Neural Architecture Search [35.362883630015354]
  Most neural network models deployed in mobile devices are tiny. However, tiny neural networks are commonly very vulnerable to attacks. Our work focuses on how to improve the robustness of tiny neural networks without seriously deteriorating of clean accuracy under mobile-level resources.
  arXiv  Detail & Related papers  (2021-02-28T00:54:09Z)
• Improving Neural Network Robustness through Neighborhood Preserving Layers [0.751016548830037]
  We demonstrate a novel neural network architecture which can incorporate such layers and also can be trained efficiently. We empirically show that our designed network architecture is more robust against state-of-art gradient descent based attacks.
  arXiv  Detail & Related papers  (2021-01-28T01:26:35Z)
• Do Wider Neural Networks Really Help Adversarial Robustness? [92.8311752980399]
  We show that the model robustness is closely related to the tradeoff between natural accuracy and perturbation stability. We propose a new Width Adjusted Regularization (WAR) method that adaptively enlarges $lambda$ on wide models.
  arXiv  Detail & Related papers  (2020-10-03T04:46:17Z)
• Rethinking Clustering for Robustness [56.14672993686335]
  ClusTR is a clustering-based and adversary-free training framework to learn robust models. textitClusTR outperforms adversarially-trained networks by up to $4%$ under strong PGD attacks.
  arXiv  Detail & Related papers  (2020-06-13T16:55:51Z)
• Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness [97.67477497115163]
  We use mode connectivity to study the adversarial robustness of deep neural networks. Our experiments cover various types of adversarial attacks applied to different network architectures and datasets. Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness.
  arXiv  Detail & Related papers  (2020-04-30T19:12:50Z)
• When Residual Learning Meets Dense Aggregation: Rethinking the Aggregation of Deep Neural Networks [57.0502745301132]
  We propose Micro-Dense Nets, a novel architecture with global residual learning and local micro-dense aggregations. Our micro-dense block can be integrated with neural architecture search based models to boost their performance.
  arXiv  Detail & Related papers  (2020-04-19T08:34:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.