RBFormer: Improve Adversarial Robustness of Transformer by Robust Bias

• URL: http://arxiv.org/abs/2309.13245v1
• Date: Sat, 23 Sep 2023 03:55:51 GMT
• Title: RBFormer: Improve Adversarial Robustness of Transformer by Robust Bias
• Authors: Hao Cheng, Jinhao Duan, Hui Li, Lyutianyang Zhang, Jiahang Cao, Ping Wang, Jize Zhang, Kaidi Xu, Renjing Xu
• Abstract summary: Introducing adversarial examples as a robustness consideration has had a profound and detrimental impact on the performance of well-established convolution-based structures. In this paper, we employ a rational structure design approach to mitigate such vulnerabilities. We introduce a novel structure called Robust Bias Transformer-based Structure (RBFormer) that shows robust superiority compared to several existing baseline structures.
• Score: 18.705151702198854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, there has been a surge of interest and attention in Transformer-based structures, such as Vision Transformer (ViT) and Vision Multilayer Perceptron (VMLP). Compared with the previous convolution-based structures, the Transformer-based structure under investigation showcases a comparable or superior performance under its distinctive attention-based input token mixer strategy. Introducing adversarial examples as a robustness consideration has had a profound and detrimental impact on the performance of well-established convolution-based structures. This inherent vulnerability to adversarial attacks has also been demonstrated in Transformer-based structures. In this paper, our emphasis lies on investigating the intrinsic robustness of the structure rather than introducing novel defense measures against adversarial attacks. To address the susceptibility to robustness issues, we employ a rational structure design approach to mitigate such vulnerabilities. Specifically, we enhance the adversarial robustness of the structure by increasing the proportion of high-frequency structural robust biases. As a result, we introduce a novel structure called Robust Bias Transformer-based Structure (RBFormer) that shows robust superiority compared to several existing baseline structures. Through a series of extensive experiments, RBFormer outperforms the original structures by a significant margin, achieving an impressive improvement of +16.12% and +5.04% across different evaluation criteria on CIFAR-10 and ImageNet-1k, respectively.

Related papers

• Isomorphic Pruning for Vision Models [56.286064975443026]
  Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. We present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures.
  arXiv  Detail & Related papers  (2024-07-05T16:14:53Z)
• On the interplay of adversarial robustness and architecture components: patches, convolution and attention [65.20660287833537]
  We study the effect of adversarial training on the interpretability of the learnt features and robustness to unseen threat models. An ablation from ResNet to ConvNeXt reveals key architectural changes leading to almost $10%$ higher $ell_infty$-robustness.
  arXiv  Detail & Related papers  (2022-09-14T22:02:32Z)
• Structural Bias for Aspect Sentiment Triplet Extraction [15.273669042985883]
  Structural bias has been exploited for aspect sentiment triplet extraction (ASTE) and led to improved performance. It is recognized that explicitly incorporating structural bias would have a negative impact on efficiency, whereas pretrained language models (PLMs) can already capture implicit structures. We propose to address the efficiency issues by using an adapter to integrate structural bias in the PLM and using a cheap-to-compute relative position structure.
  arXiv  Detail & Related papers  (2022-09-02T05:02:18Z)
• Deeper Insights into ViTs Robustness towards Common Corruptions [82.79764218627558]
  We investigate how CNN-like architectural designs and CNN-based data augmentation strategies impact on ViTs' robustness towards common corruptions. We demonstrate that overlapping patch embedding and convolutional Feed-Forward Network (FFN) boost performance on robustness. We also introduce a novel conditional method enabling input-varied augmentations from two angles.
  arXiv  Detail & Related papers  (2022-04-26T08:22:34Z)
• Clustering Effect of (Linearized) Adversarial Robust Models [60.25668525218051]
  We propose a novel understanding of adversarial robustness and apply it on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy.
  arXiv  Detail & Related papers  (2021-11-25T05:51:03Z)
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques [170.3297213957074]
  Deep neural networks (DNNs) are vulnerable to adversarial noises. There are no comprehensive studies of how architecture design and training techniques affect robustness. We propose the first comprehensiveness investigation benchmark on ImageNet.
  arXiv  Detail & Related papers  (2021-09-11T08:01:14Z)
• Understanding Structural Vulnerability in Graph Convolutional Networks [27.602802961213236]
  Graph Convolutional Networks (GCNs) are vulnerable to adversarial attacks on the graph structure. We show that structural adversarial examples can be attributed to the non-robust aggregation scheme of GCNs. We show that adopting the aggregation scheme with a high breakdown point could significantly enhance the robustness of GCNs against structural attacks.
  arXiv  Detail & Related papers  (2021-08-13T15:07:44Z)
• Where Does the Robustness Come from? A Study of the Transformation-based Ensemble Defence [12.973226757056462]
  It is not clear whether the robustness improvement is a result of transformation or ensemble. We conduct experiments to show that 1) the transferability of adversarial examples exists among the models trained on data records after different reversible transformations; 2) the robustness gained through transformation-based ensemble is limited; and 3) this limited robustness is mainly from the irreversible transformations rather than the ensemble of a number of models.
  arXiv  Detail & Related papers  (2020-09-28T02:55:56Z)
• Towards a Theoretical Understanding of the Robustness of Variational Autoencoders [82.68133908421792]
  We make inroads into understanding the robustness of Variational Autoencoders (VAEs) to adversarial attacks and other input perturbations. We develop a novel criterion for robustness in probabilistic models: $r$-robustness. We show that VAEs trained using disentangling methods score well under our robustness metrics.
  arXiv  Detail & Related papers  (2020-07-14T21:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.