Benign Overfitting in Adversarial Training for Vision Transformers

• URL: http://arxiv.org/abs/2604.19724v1
• Date: Tue, 21 Apr 2026 17:48:51 GMT
• Title: Benign Overfitting in Adversarial Training for Vision Transformers
• Abstract summary: A common empirical defense strategy is adversarial training, yet the theoretical underpinnings of its robustness in ViTs remain largely unexplored.<n>We show that, when trained under a signal-to-noise ratio that satisfies a certain condition and within a moderate budget, adversarial training enables ViTs to achieve nearly zero robust training loss and robust generalization error under certain regimes.
• Score: 26.33093636776093
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the remarkable success of Vision Transformers (ViTs) across a wide range of vision tasks, recent studies have revealed that they remain vulnerable to adversarial examples, much like Convolutional Neural Networks (CNNs). A common empirical defense strategy is adversarial training, yet the theoretical underpinnings of its robustness in ViTs remain largely unexplored. In this work, we present the first theoretical analysis of adversarial training under simplified ViT architectures. We show that, when trained under a signal-to-noise ratio that satisfies a certain condition and within a moderate perturbation budget, adversarial training enables ViTs to achieve nearly zero robust training loss and robust generalization error under certain regimes. Remarkably, this leads to strong generalization even in the presence of overfitting, a phenomenon known as \emph{benign overfitting}, previously only observed in CNNs (with adversarial training). Experiments on both synthetic and real-world datasets further validate our theoretical findings.

Related papers

• Generalist++: A Meta-learning Framework for Mitigating Trade-off in Adversarial Training [105.74524789405514]
  adversarial training (AT) is currently the most effective defense against neural networks.<n>We propose to partition the overall generalization goal into multiple sub-tasks, each assigned to a dedicated base learner.<n>In the later stages of training, we interpolate their parameters to form a knowledgeable global learner.<n>We term this framework Generalist and introduce three variants tailored to different application scenarios.
  arXiv  Detail & Related papers  (2025-10-15T09:47:54Z)
• Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data [38.44734564565478]
  We provide a theoretical understanding of adversarial examples and adversarial training algorithms from the perspective of feature learning theory.<n>We show that the adversarial training method can provably strengthen the robust feature learning and suppress the non-robust feature learning.
  arXiv  Detail & Related papers  (2024-10-11T03:59:49Z)
• Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization [88.5582111768376]
  We study the optimization of a Transformer composed of a self-attention layer with softmax followed by a fully connected layer under gradient descent on a certain data distribution model. Our results establish a sharp condition that can distinguish between the small test error phase and the large test error regime, based on the signal-to-noise ratio in the data model.
  arXiv  Detail & Related papers  (2024-09-28T13:24:11Z)
• Evolution of Neural Tangent Kernels under Benign and Adversarial Training [109.07737733329019]
  We study the evolution of the empirical Neural Tangent Kernel (NTK) under standard and adversarial training. We find under adversarial training, the empirical NTK rapidly converges to a different kernel (and feature map) than standard training. This new kernel provides adversarial robustness, even when non-robust training is performed on top of it.
  arXiv  Detail & Related papers  (2022-10-21T15:21:15Z)
• Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem [15.033609196411517]
  Vision Transformers (ViTs) surpass the Convolutional Neural Networks (CNNs) under some perturbations. We introduce a principled and unified theoretical framework to investigate ViT's robustness.
  arXiv  Detail & Related papers  (2022-08-01T14:50:29Z)
• Towards Efficient Adversarial Training on Vision Transformers [41.6396577241957]
  Adversarial training is one of the most effective ways to accomplish robust CNNs. We propose an efficient Attention Guided Adversarial Training mechanism. With only 65% of the fast adversarial training time, we match the state-of-the-art results on the challenging ImageNet benchmark.
  arXiv  Detail & Related papers  (2022-07-21T14:23:50Z)
• Latent Boundary-guided Adversarial Training [61.43040235982727]
  Adrial training is proved to be the most effective strategy that injects adversarial examples into model training. We propose a novel adversarial training framework called LAtent bounDary-guided aDvErsarial tRaining.
  arXiv  Detail & Related papers  (2022-06-08T07:40:55Z)
• Enhancing Adversarial Training with Feature Separability [52.39305978984573]
  We introduce a new concept of adversarial training graph (ATG) with which the proposed adversarial training with feature separability (ATFS) enables to boost the intra-class feature similarity and increase inter-class feature variance. Through comprehensive experiments, we demonstrate that the proposed ATFS framework significantly improves both clean and robust performance.
  arXiv  Detail & Related papers  (2022-05-02T04:04:23Z)
• Finding Dynamics Preserving Adversarial Winning Tickets [11.05616199881368]
  Pruning methods have been considered in adversarial context to reduce model capacity and improve adversarial robustness simultaneously in training. Existing adversarial pruning methods generally mimic the classical pruning methods for natural training, which follow the three-stage 'training-pruning-fine-tuning' pipelines. We show empirical evidences that AWT preserves the dynamics of adversarial training and achieve equal performance as dense adversarial training.
  arXiv  Detail & Related papers  (2022-02-14T05:34:24Z)
• On the Adversarial Robustness of Visual Transformers [129.29523847765952]
  This work provides the first and comprehensive study on the robustness of vision transformers (ViTs) against adversarial perturbations. Tested on various white-box and transfer attack settings, we find that ViTs possess better adversarial robustness when compared with convolutional neural networks (CNNs)
  arXiv  Detail & Related papers  (2021-03-29T14:48:24Z)
• Towards Understanding the Dynamics of the First-Order Adversaries [40.54670072901657]
  An acknowledged weakness of neural networks is their vulnerability to adversarial perturbations to the inputs. One of the most popular defense mechanisms is to maximize the loss over the constrained perturbations on the inputs using projected ascent and minimize over weights. We investigate the non-concave landscape of the adversaries for a two-layer neural network with a quadratic loss.
  arXiv  Detail & Related papers  (2020-10-20T22:20:53Z)
• Overfitting in adversarially robust deep learning [86.11788847990783]
  We show that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting.
  arXiv  Detail & Related papers  (2020-02-26T15:40:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.