The Sword of Damocles in ViTs: Computational Redundancy Amplifies Adversarial Transferability

• URL: http://arxiv.org/abs/2504.10804v1
• Date: Tue, 15 Apr 2025 01:59:47 GMT
• Title: The Sword of Damocles in ViTs: Computational Redundancy Amplifies Adversarial Transferability
• Authors: Jiani Liu, Zhiyuan Wang, Zeliang Zhang, Chao Huang, Susan Liang, Yunlong Tang, Chenliang Xu,
• Abstract summary: We investigate the role of computational redundancy in Vision Transformers (ViTs) and its impact on adversarial transferability.<n>We identify two forms of redundancy, including the data-level and model-level, that can be harnessed to amplify attack effectiveness.<n>Building on this insight, we design a suite of techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and test-time adversarial training.
• Score: 38.32538271219404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformers (ViTs) have demonstrated impressive performance across a range of applications, including many safety-critical tasks. However, their unique architectural properties raise new challenges and opportunities in adversarial robustness. In particular, we observe that adversarial examples crafted on ViTs exhibit higher transferability compared to those crafted on CNNs, suggesting that ViTs contain structural characteristics favorable for transferable attacks. In this work, we investigate the role of computational redundancy in ViTs and its impact on adversarial transferability. Unlike prior studies that aim to reduce computation for efficiency, we propose to exploit this redundancy to improve the quality and transferability of adversarial examples. Through a detailed analysis, we identify two forms of redundancy, including the data-level and model-level, that can be harnessed to amplify attack effectiveness. Building on this insight, we design a suite of techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and test-time adversarial training. Extensive experiments on the ImageNet-1k dataset validate the effectiveness of our approach, showing that our methods significantly outperform existing baselines in both transferability and generality across diverse model architectures.

Related papers

• Attacking Transformers with Feature Diversity Adversarial Perturbation [19.597912600568026]
  We present a label-free white-box attack approach for ViT-based models that exhibits strong transferability to various black box models. Our inspiration comes from the feature collapse phenomenon in ViTs, where the critical attention mechanism overly depends on the low-frequency component of features.
  arXiv  Detail & Related papers  (2024-03-10T00:55:58Z)
• Bag of Tricks to Boost Adversarial Transferability [5.803095119348021]
  adversarial examples generated under the white-box setting often exhibit low transferability across different models. In this work, we find that several tiny changes in the existing adversarial attacks can significantly affect the attack performance. Based on careful studies of existing adversarial attacks, we propose a bag of tricks to enhance adversarial transferability.
  arXiv  Detail & Related papers  (2024-01-16T17:42:36Z)
• CT-GAT: Cross-Task Generative Adversarial Attack based on Transferability [24.272384832200522]
  We propose a novel approach that directly constructs adversarial examples by extracting transferable features across various tasks. Specifically, we train a sequence-to-sequence generative model named CT-GAT using adversarial sample data collected from multiple tasks to acquire universal adversarial features. Results demonstrate that our method achieves superior attack performance with small cost.
  arXiv  Detail & Related papers  (2023-10-22T11:00:04Z)
• Demystify Transformers & Convolutions in Modern Image Deep Networks [80.16624587948368]
  This paper aims to identify the real gains of popular convolution and attention operators through a detailed study.<n>We find that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach.<n>Various STMs are integrated into this unified framework for comprehensive comparative analysis.
  arXiv  Detail & Related papers  (2022-11-10T18:59:43Z)
• Deeper Insights into ViTs Robustness towards Common Corruptions [82.79764218627558]
  We investigate how CNN-like architectural designs and CNN-based data augmentation strategies impact on ViTs' robustness towards common corruptions. We demonstrate that overlapping patch embedding and convolutional Feed-Forward Network (FFN) boost performance on robustness. We also introduce a novel conditional method enabling input-varied augmentations from two angles.
  arXiv  Detail & Related papers  (2022-04-26T08:22:34Z)
• Towards Transferable Adversarial Attacks on Vision Transformers [110.55845478440807]
  Vision transformers (ViTs) have demonstrated impressive performance on a series of computer vision tasks, yet they still suffer from adversarial examples. We introduce a dual attack framework, which contains a Pay No Attention (PNA) attack and a PatchOut attack, to improve the transferability of adversarial samples across different ViTs.
  arXiv  Detail & Related papers  (2021-09-09T11:28:25Z)
• TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation [54.61786380919243]
  Unsupervised domain adaptation (UDA) aims to transfer the knowledge learnt from a labeled source domain to an unlabeled target domain. Previous work is mainly built upon convolutional neural networks (CNNs) to learn domain-invariant representations. With the recent exponential increase in applying Vision Transformer (ViT) to vision tasks, the capability of ViT in adapting cross-domain knowledge remains unexplored in the literature.
  arXiv  Detail & Related papers  (2021-08-12T22:37:43Z)
• Enhancing Adversarial Robustness via Test-time Transformation Ensembling [51.51139269928358]
  We show how equipping models with Test-time Transformation Ensembling can work as a reliable defense against adversarial attacks. We show that TTE consistently improves model robustness against a variety of powerful attacks without any need for re-training.
  arXiv  Detail & Related papers  (2021-07-29T15:32:35Z)
• TREND: Transferability based Robust ENsemble Design [6.663641564969944]
  We study the effect of network architecture, input, weight and activation quantization on transferability of adversarial samples. We show that transferability is significantly hampered by input quantization between source and target. We propose a new state-of-the-art ensemble attack to combat this.
  arXiv  Detail & Related papers  (2020-08-04T13:38:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.