ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers

• URL: http://arxiv.org/abs/2508.12384v1
• Date: Sun, 17 Aug 2025 14:47:31 GMT
• Title: ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers
• Authors: Hanwen Cao, Haobo Lu, Xiaosen Wang, Kun He,
• Abstract summary: We propose ViT-EnsembleAttack to boost overall generalization of ensemble models and reduce the risk of adversarial overfitting.<n>ViT-EnsembleAttack significantly enhances the adversarial transferability of ensemble-based attacks on ViTs, outperforming existing methods by a substantial margin.
• Score: 12.042884657815845
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensemble-based attacks have been proven to be effective in enhancing adversarial transferability by aggregating the outputs of models with various architectures. However, existing research primarily focuses on refining ensemble weights or optimizing the ensemble path, overlooking the exploration of ensemble models to enhance the transferability of adversarial attacks. To address this gap, we propose applying adversarial augmentation to the surrogate models, aiming to boost overall generalization of ensemble models and reduce the risk of adversarial overfitting. Meanwhile, observing that ensemble Vision Transformers (ViTs) gain less attention, we propose ViT-EnsembleAttack based on the idea of model adversarial augmentation, the first ensemble-based attack method tailored for ViTs to the best of our knowledge. Our approach generates augmented models for each surrogate ViT using three strategies: Multi-head dropping, Attention score scaling, and MLP feature mixing, with the associated parameters optimized by Bayesian optimization. These adversarially augmented models are ensembled to generate adversarial examples. Furthermore, we introduce Automatic Reweighting and Step Size Enlargement modules to boost transferability. Extensive experiments demonstrate that ViT-EnsembleAttack significantly enhances the adversarial transferability of ensemble-based attacks on ViTs, outperforming existing methods by a substantial margin. Code is available at https://github.com/Trustworthy-AI-Group/TransferAttack.

Related papers

• Proxy Robustness in Vision Language Models is Effortlessly Transferable [13.390016978827163]
  A pivotal technique for improving the defense of deep models, adversarial robustness transfer via distillation has demonstrated remarkable success in conventional image classification tasks.<n>We bridge this gap by revealing an interesting phenomenon: vanilla CLIP (without adversarial training) exhibits intrinsic defensive capabilities against adversarial examples.<n>We formally define this as proxy adversarial robustness, and naturally propose a Heterogeneous Proxy Transfer framework.
  arXiv  Detail & Related papers  (2026-01-19T09:23:11Z)
• Boosting Adversarial Transferability via Ensemble Non-Attention [12.414747362069457]
  We design a novel ensemble attack, NAMEA, which integrates the gradients from the non-attention areas of ensemble models into the iterative gradient optimization process.<n> NAMEA outperforms AdaEA and SMER, the state-of-the-art ensemble attacks by an average of 15.0% and 9.6%, respectively.
  arXiv  Detail & Related papers  (2025-11-12T03:25:25Z)
• A Simple DropConnect Approach to Transfer-based Targeted Attack [43.039945949426546]
  We study the problem of transfer-based black-box attack, where adversarial samples generated using a single surrogate model are directly applied to target models.<n>We propose to Mitigate perturbation Co-adaptation by DropConnect to enhance transferability.<n>In the challenging scenario of transferring from a CNN-based model to Transformer-based models, MCD achieves 13% higher average ASRs compared with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-24T12:29:23Z)
• Boosting Adversarial Transferability with Spatial Adversarial Alignment [56.97809949196889]
  Deep neural networks are vulnerable to adversarial examples that exhibit transferability across various models.<n>We propose a technique that employs an alignment loss and leverages a witness model to fine-tune the surrogate model.<n>Experiments on various architectures on ImageNet show that aligned surrogate models based on SAA can provide higher transferable adversarial examples.
  arXiv  Detail & Related papers  (2025-01-02T02:35:47Z)
• Adversarial Robustness through Dynamic Ensemble Learning [0.0]
  Adversarial attacks pose a significant threat to the reliability of pre-trained language models (PLMs)<n>This paper presents Adversarial Robustness through Dynamic Ensemble Learning (ARDEL), a novel scheme designed to enhance the robustness of PLMs against such attacks.
  arXiv  Detail & Related papers  (2024-12-20T05:36:19Z)
• Transferable Adversarial Attacks on SAM and Its Downstream Models [87.23908485521439]
  This paper explores the feasibility of adversarial attacking various downstream models fine-tuned from the segment anything model (SAM)<n>To enhance the effectiveness of the adversarial attack towards models fine-tuned on unknown datasets, we propose a universal meta-initialization (UMI) algorithm.
  arXiv  Detail & Related papers  (2024-10-26T15:04:04Z)
• Enhancing Adversarial Transferability with Adversarial Weight Tuning [50.01825144613307]
  adversarial examples (AEs) mislead the model while appearing benign to human observers.<n>AWT is a data-free tuning method that combines gradient-based and model-based attack methods to enhance the transferability of AEs.
  arXiv  Detail & Related papers  (2024-08-18T13:31:26Z)
• Ensemble Adversarial Defense via Integration of Multiple Dispersed Low Curvature Models [7.8245455684263545]
  In this work, we aim to enhance ensemble diversity by reducing attack transferability. We identify second-order gradients, which depict the loss curvature, as a key factor in adversarial robustness. We introduce a novel regularizer to train multiple more-diverse low-curvature network models.
  arXiv  Detail & Related papers  (2024-03-25T03:44:36Z)
• FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers [72.83770102062141]
  Vision Transformer (ViT) model has gradually become mainstream in various computer vision tasks.<n>Existing large models tend to prioritize performance during training, potentially neglecting the robustness.<n>We develop novel LNLoRA module, incorporating a learnable layer normalization before the conventional LoRA module.<n>We propose the FullLoRA framework by integrating the learnable LNLoRA modules into all key components of ViT-based models.
  arXiv  Detail & Related papers  (2024-01-03T14:08:39Z)
• Enhancing Adversarial Attacks: The Similar Target Method [6.293148047652131]
  adversarial examples pose a threat to deep neural networks' applications. Deep neural networks are vulnerable to adversarial examples, posing a threat to the models' applications and raising security concerns. We propose a similar targeted attack method named Similar Target(ST)
  arXiv  Detail & Related papers  (2023-08-21T14:16:36Z)
• An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability [26.39964737311377]
  We propose an adaptive ensemble attack, dubbed AdaEA, to adaptively control the fusion of the outputs from each model. We achieve considerable improvement over the existing ensemble attacks on various datasets.
  arXiv  Detail & Related papers  (2023-08-05T15:12:36Z)
• Improving Transferability of Adversarial Examples via Bayesian Attacks [68.90574788107442]
  adversarial examples allows for the attack on unknown deep neural networks (DNNs)<n>In this paper, we improve the transferability of adversarial examples by incorporating the Bayesian formulation into both the model parameters and model input.<n>Experiments demonstrate that our method achieves a new state-of-the-art in transfer-based attacks.
  arXiv  Detail & Related papers  (2023-07-21T03:43:07Z)
• Deeper Insights into ViTs Robustness towards Common Corruptions [82.79764218627558]
  We investigate how CNN-like architectural designs and CNN-based data augmentation strategies impact on ViTs' robustness towards common corruptions. We demonstrate that overlapping patch embedding and convolutional Feed-Forward Network (FFN) boost performance on robustness. We also introduce a novel conditional method enabling input-varied augmentations from two angles.
  arXiv  Detail & Related papers  (2022-04-26T08:22:34Z)
• On Improving Adversarial Transferability of Vision Transformers [97.17154635766578]
  Vision transformers (ViTs) process input images as sequences of patches via self-attention. We study the adversarial feature space of ViT models and their transferability. We introduce two novel strategies specific to the architecture of ViT models.
  arXiv  Detail & Related papers  (2021-06-08T08:20:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.