Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem

• URL: http://arxiv.org/abs/2208.00906v1
• Date: Mon, 1 Aug 2022 14:50:29 GMT
• Title: Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem
• Abstract summary: Vision Transformers (ViTs) surpass the Convolutional Neural Networks (CNNs) under some perturbations. We introduce a principled and unified theoretical framework to investigate ViT's robustness.
• Score: 15.033609196411517
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent research on the robustness of deep learning has shown that Vision Transformers (ViTs) surpass the Convolutional Neural Networks (CNNs) under some perturbations, e.g., natural corruption, adversarial attacks, etc. Some papers argue that the superior robustness of ViT comes from the segmentation of its input images; others say that the Multi-head Self-Attention (MSA) is the key to preserving the robustness. In this paper, we aim to introduce a principled and unified theoretical framework to investigate such an argument on ViT's robustness. We first theoretically prove that, unlike Transformers in Natural Language Processing, ViTs are Lipschitz continuous. Then we theoretically analyze the adversarial robustness of ViTs from the perspective of the Cauchy Problem, via which we can quantify how the robustness propagates through layers. We demonstrate that the first and last layers are the critical factors to affect the robustness of ViTs. Furthermore, based on our theory, we empirically show that unlike the claims from existing research, MSA only contributes to the adversarial robustness of ViTs under weak adversarial attacks, e.g., FGSM, and surprisingly, MSA actually comprises the model's adversarial robustness under stronger attacks, e.g., PGD attacks.

Related papers

• Benign Overfitting in Adversarial Training for Vision Transformers [26.33093636776093]
  A common empirical defense strategy is adversarial training, yet the theoretical underpinnings of its robustness in ViTs remain largely unexplored.<n>We show that, when trained under a signal-to-noise ratio that satisfies a certain condition and within a moderate budget, adversarial training enables ViTs to achieve nearly zero robust training loss and robust generalization error under certain regimes.
  arXiv  Detail & Related papers  (2026-04-21T17:48:51Z)
• Vision Transformers Need More Than Registers [70.42157905484765]
  Vision Transformers (ViTs) provide general-purpose representations for diverse downstream tasks.<n> artifacts in ViTs are widely observed across different supervision paradigms and downstream tasks.<n>We conclude that these artifacts originate from a lazy aggregation behavior.
  arXiv  Detail & Related papers  (2026-02-25T20:42:35Z)
• The Sword of Damocles in ViTs: Computational Redundancy Amplifies Adversarial Transferability [38.32538271219404]
  We investigate the role of computational redundancy in Vision Transformers (ViTs) and its impact on adversarial transferability. We identify two forms of redundancy, including the data-level and model-level, that can be harnessed to amplify attack effectiveness. Building on this insight, we design a suite of techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and test-time adversarial training.
  arXiv  Detail & Related papers  (2025-04-15T01:59:47Z)
• Mechanistic Understandings of Representation Vulnerabilities and Engineering Robust Vision Transformers [1.1187085721899017]
  We study the sources of known representation vulnerabilities of vision transformers (ViT), where perceptually identical images can have very different representations. We develop NeuroShield-ViT, a novel defense mechanism that strategically neutralizes vulnerable neurons in earlier layers to prevent the cascade of adversarial effects. Our results shed new light on how adversarial effects propagate through ViT layers, while providing a promising approach to enhance the robustness of vision transformers against adversarial attacks.
  arXiv  Detail & Related papers  (2025-02-07T05:58:16Z)
• MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness [31.603115393528746]
  Building robust Vision Transformers (ViTs) is highly dependent on dedicated Adversarial Training (AT) strategies. We provide a novel theoretical Mutual Information (MI) analysis in its autoencoder-based self-supervised pre-training. We propose a masked autoencoder-based pre-training method, MIMIR, that employs an MI penalty to facilitate the adversarial training of ViTs.
  arXiv  Detail & Related papers  (2023-12-08T10:50:02Z)
• Improving Interpretation Faithfulness for Vision Transformers [42.86486715574245]
  Vision Transformers (ViTs) have achieved state-of-the-art performance for various vision tasks. ViTs suffer from issues with explanation faithfulness, as their focal points are fragile to adversarial attacks. We propose a rigorous approach to mitigate these issues by introducing Faithful ViTs (FViTs)
  arXiv  Detail & Related papers  (2023-11-29T18:51:21Z)
• Self-Ensembling Vision Transformer (SEViT) for Robust Medical Image Classification [4.843654097048771]
  Vision Transformers (ViT) are competing to replace Convolutional Neural Networks (CNN) for various computer vision tasks in medical imaging. Recent works have shown that ViTs are also susceptible to such attacks and suffer significant performance degradation under attack. We propose a novel self-ensembling method to enhance the robustness of ViT in the presence of adversarial attacks.
  arXiv  Detail & Related papers  (2022-08-04T19:02:24Z)
• Deeper Insights into ViTs Robustness towards Common Corruptions [82.79764218627558]
  We investigate how CNN-like architectural designs and CNN-based data augmentation strategies impact on ViTs' robustness towards common corruptions. We demonstrate that overlapping patch embedding and convolutional Feed-Forward Network (FFN) boost performance on robustness. We also introduce a novel conditional method enabling input-varied augmentations from two angles.
  arXiv  Detail & Related papers  (2022-04-26T08:22:34Z)
• Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations? [21.32962679185015]
  Vision transformers (ViTs) have recently set off a new wave in neural architecture design thanks to their record-breaking performance in vision tasks. Recent works show that ViTs are more robust against adversarial attacks as compared with convolutional neural networks (CNNs) We propose a dedicated attack framework, dubbed Patch-Fool, that fools the self-attention mechanism by attacking its basic component.
  arXiv  Detail & Related papers  (2022-03-16T04:45:59Z)
• Towards Transferable Adversarial Attacks on Vision Transformers [110.55845478440807]
  Vision transformers (ViTs) have demonstrated impressive performance on a series of computer vision tasks, yet they still suffer from adversarial examples. We introduce a dual attack framework, which contains a Pay No Attention (PNA) attack and a PatchOut attack, to improve the transferability of adversarial samples across different ViTs.
  arXiv  Detail & Related papers  (2021-09-09T11:28:25Z)
• Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
  We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs. We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
  arXiv  Detail & Related papers  (2021-06-21T21:42:08Z)
• Adversarial Visual Robustness by Causal Intervention [56.766342028800445]
  Adversarial training is the de facto most promising defense against adversarial examples. Yet, its passive nature inevitably prevents it from being immune to unknown attackers. We provide a causal viewpoint of adversarial vulnerability: the cause is the confounder ubiquitously existing in learning.
  arXiv  Detail & Related papers  (2021-06-17T14:23:54Z)
• On Improving Adversarial Transferability of Vision Transformers [97.17154635766578]
  Vision transformers (ViTs) process input images as sequences of patches via self-attention. We study the adversarial feature space of ViT models and their transferability. We introduce two novel strategies specific to the architecture of ViT models.
  arXiv  Detail & Related papers  (2021-06-08T08:20:38Z)
• On the Adversarial Robustness of Visual Transformers [129.29523847765952]
  This work provides the first and comprehensive study on the robustness of vision transformers (ViTs) against adversarial perturbations. Tested on various white-box and transfer attack settings, we find that ViTs possess better adversarial robustness when compared with convolutional neural networks (CNNs)
  arXiv  Detail & Related papers  (2021-03-29T14:48:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.