Vision Transformers: the threat of realistic adversarial patches

• URL: http://arxiv.org/abs/2509.21084v1
• Date: Thu, 25 Sep 2025 12:36:25 GMT
• Title: Vision Transformers: the threat of realistic adversarial patches
• Authors: Kasper Cools, Clara Maathuis, Alexander M. van Oers, Claudia S. Hübner, Nikos Deligiannis, Marijke Vandewal, Geert De Cubber,
• Abstract summary: Vision Transformers (ViTs) have gained significant traction in modern machine learning.<n>ViTs remain vulnerable to evasion attacks, particularly to adversarial patches.<n>This study investigates the transferability of adversarial attack techniques used in CNNs when applied to ViT classification models.
• Score: 48.03238826812818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The increasing reliance on machine learning systems has made their security a critical concern. Evasion attacks enable adversaries to manipulate the decision-making processes of AI systems, potentially causing security breaches or misclassification of targets. Vision Transformers (ViTs) have gained significant traction in modern machine learning due to increased 1) performance compared to Convolutional Neural Networks (CNNs) and 2) robustness against adversarial perturbations. However, ViTs remain vulnerable to evasion attacks, particularly to adversarial patches, unique patterns designed to manipulate AI classification systems. These vulnerabilities are investigated by designing realistic adversarial patches to cause misclassification in person vs. non-person classification tasks using the Creases Transformation (CT) technique, which adds subtle geometric distortions similar to those occurring naturally when wearing clothing. This study investigates the transferability of adversarial attack techniques used in CNNs when applied to ViT classification models. Experimental evaluation across four fine-tuned ViT models on a binary person classification task reveals significant vulnerability variations: attack success rates ranged from 40.04% (google/vit-base-patch16-224-in21k) to 99.97% (facebook/dino-vitb16), with google/vit-base-patch16-224 achieving 66.40% and facebook/dinov3-vitb16 reaching 65.17%. These results confirm the cross-architectural transferability of adversarial patches from CNNs to ViTs, with pre-training dataset scale and methodology strongly influencing model resilience to adversarial attacks.

Related papers

• Breaking the Illusion of Security via Interpretation: Interpretable Vision Transformer Systems under Attack [23.690939122119723]
  Vision transformer (ViT) models, when coupled with interpretation models, are regarded as secure and challenging to deceive.<n>This study investigates the vulnerability of transformer models to adversarial attacks, even when combined with interpretation models.<n>We propose an attack called "AdViT" that generates adversarial examples capable of misleading both a given transformer model and its coupled interpretation model.
  arXiv  Detail & Related papers  (2025-07-18T05:11:11Z)
• ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer [8.71614629110101]
  We propose ViTGuard as a general detection method for defending Vision Transformer (ViT) models against adversarial attacks. ViTGuard uses a Masked Autoencoder (MAE) model to recover randomly masked patches from the unmasked regions. threshold-based detectors leverage distinctive ViT features, including attention maps and classification (token representations) token representations, to distinguish between normal and adversarial samples.
  arXiv  Detail & Related papers  (2024-09-20T18:11:56Z)
• Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers [95.22517830759193]
  This paper studies the transferability of such an adversarial vulnerability from a pre-trained ViT model to downstream tasks. We show that DTA achieves an average attack success rate (ASR) exceeding 90%, surpassing existing methods by a huge margin.
  arXiv  Detail & Related papers  (2024-08-03T08:07:03Z)
• Black-box Adversarial Transferability: An Empirical Study in Cybersecurity Perspective [0.0]
  In adversarial machine learning, malicious users try to fool the deep learning model by inserting adversarial perturbation inputs into the model during its training or testing phase. We empirically test the black-box adversarial transferability phenomena in cyber attack detection systems. The results indicate that any deep learning model is highly susceptible to adversarial attacks, even if the attacker does not have access to the internal details of the target model.
  arXiv  Detail & Related papers  (2024-04-15T06:56:28Z)
• The Efficacy of Transformer-based Adversarial Attacks in Security Domains [0.7156877824959499]
  We evaluate the robustness of transformers to adversarial samples for system defenders and their adversarial strength for system attackers. Our work emphasizes the importance of studying transformer architectures for attacking and defending models in security domains.
  arXiv  Detail & Related papers  (2023-10-17T21:45:23Z)
• Self-Ensembling Vision Transformer (SEViT) for Robust Medical Image Classification [4.843654097048771]
  Vision Transformers (ViT) are competing to replace Convolutional Neural Networks (CNN) for various computer vision tasks in medical imaging. Recent works have shown that ViTs are also susceptible to such attacks and suffer significant performance degradation under attack. We propose a novel self-ensembling method to enhance the robustness of ViT in the presence of adversarial attacks.
  arXiv  Detail & Related papers  (2022-08-04T19:02:24Z)
• Towards Transferable Adversarial Attacks on Vision Transformers [110.55845478440807]
  Vision transformers (ViTs) have demonstrated impressive performance on a series of computer vision tasks, yet they still suffer from adversarial examples. We introduce a dual attack framework, which contains a Pay No Attention (PNA) attack and a PatchOut attack, to improve the transferability of adversarial samples across different ViTs.
  arXiv  Detail & Related papers  (2021-09-09T11:28:25Z)
• Evaluating the Robustness of Semantic Segmentation for Autonomous Driving against Real-World Adversarial Patch Attacks [62.87459235819762]
  In a real-world scenario like autonomous driving, more attention should be devoted to real-world adversarial examples (RWAEs) This paper presents an in-depth evaluation of the robustness of popular SS models by testing the effects of both digital and real-world adversarial patches.
  arXiv  Detail & Related papers  (2021-08-13T11:49:09Z)
• Adaptive Feature Alignment for Adversarial Training [56.17654691470554]
  CNNs are typically vulnerable to adversarial attacks, which pose a threat to security-sensitive applications. We propose the adaptive feature alignment (AFA) to generate features of arbitrary attacking strengths. Our method is trained to automatically align features of arbitrary attacking strength.
  arXiv  Detail & Related papers  (2021-05-31T17:01:05Z)
• On the Adversarial Robustness of Visual Transformers [129.29523847765952]
  This work provides the first and comprehensive study on the robustness of vision transformers (ViTs) against adversarial perturbations. Tested on various white-box and transfer attack settings, we find that ViTs possess better adversarial robustness when compared with convolutional neural networks (CNNs)
  arXiv  Detail & Related papers  (2021-03-29T14:48:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.