TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization

• URL: http://arxiv.org/abs/2505.19613v1
• Date: Mon, 26 May 2025 07:30:00 GMT
• Title: TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization
• Authors: Amira Guesmi, Bassem Ouni, Muhammad Shafique,
• Abstract summary: Adrial transferability remains a critical challenge in evaluating the robustness of deep neural networks.<n>We introduce textbfTESSER -- a novel adversarial attack framework that enhances transferability via two key strategies.<n>Experiments on ImageNet across 12 diverse architectures demonstrate that TESSER achieves +10.9% higher attack succes rate (ASR) on CNNs and +7.2% on ViTs.
• Score: 3.962831477787584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial transferability remains a critical challenge in evaluating the robustness of deep neural networks. In security-critical applications, transferability enables black-box attacks without access to model internals, making it a key concern for real-world adversarial threat assessment. While Vision Transformers (ViTs) have demonstrated strong adversarial performance, existing attacks often fail to transfer effectively across architectures, especially from ViTs to Convolutional Neural Networks (CNNs) or hybrid models. In this paper, we introduce \textbf{TESSER} -- a novel adversarial attack framework that enhances transferability via two key strategies: (1) \textit{Feature-Sensitive Gradient Scaling (FSGS)}, which modulates gradients based on token-wise importance derived from intermediate feature activations, and (2) \textit{Spectral Smoothness Regularization (SSR)}, which suppresses high-frequency noise in perturbations using a differentiable Gaussian prior. These components work in tandem to generate perturbations that are both semantically meaningful and spectrally smooth. Extensive experiments on ImageNet across 12 diverse architectures demonstrate that TESSER achieves +10.9\% higher attack succes rate (ASR) on CNNs and +7.2\% on ViTs compared to the state-of-the-art Adaptive Token Tuning (ATT) method. Moreover, TESSER significantly improves robustness against defended models, achieving 53.55\% ASR on adversarially trained CNNs. Qualitative analysis shows strong alignment between TESSER's perturbations and salient visual regions identified via Grad-CAM, while frequency-domain analysis reveals a 12\% reduction in high-frequency energy, confirming the effectiveness of spectral regularization.

Related papers

• Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability [38.32538271219404]
  We investigate the role of computational redundancy in Vision Transformers (ViTs) and its impact on adversarial transferability.<n>We identify two forms of redundancy, including the data-level and model-level, that can be harnessed to amplify attack effectiveness.<n>Building on this insight, we design a suite of techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and test-time adversarial training.
  arXiv  Detail & Related papers  (2025-04-15T01:59:47Z)
• BHViT: Binarized Hybrid Vision Transformer [53.38894971164072]
  Model binarization has made significant progress in enabling real-time and energy-efficient computation for convolutional neural networks (CNN)<n>We propose BHViT, a binarization-friendly hybrid ViT architecture and its full binarization model with the guidance of three important observations.<n>Our proposed algorithm achieves SOTA performance among binary ViT methods.
  arXiv  Detail & Related papers  (2025-03-04T08:35:01Z)
• Mechanistic Understandings of Representation Vulnerabilities and Engineering Robust Vision Transformers [1.1187085721899017]
  We study the sources of known representation vulnerabilities of vision transformers (ViT), where perceptually identical images can have very different representations.<n>We develop NeuroShield-ViT, a novel defense mechanism that strategically neutralizes vulnerable neurons in earlier layers to prevent the cascade of adversarial effects.<n>Our results shed new light on how adversarial effects propagate through ViT layers, while providing a promising approach to enhance the robustness of vision transformers against adversarial attacks.
  arXiv  Detail & Related papers  (2025-02-07T05:58:16Z)
• SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers [9.100671508333724]
  Vision transformers (ViTs) have become essential backbones in advanced computer vision applications and multi-modal foundation models.<n>Despite their strengths, ViTs remain vulnerable to adversarial perturbations, comparable to or even exceeding the vulnerability of convolutional neural networks (CNNs)<n>This paper mitigates adversarial overfitting in ViTs through a novel, layer-selective fine-tuning approach: SAFER.
  arXiv  Detail & Related papers  (2025-01-02T20:37:14Z)
• Towards Evaluating the Robustness of Visual State Space Models [63.14954591606638]
  Vision State Space Models (VSSMs) have demonstrated remarkable performance in visual perception tasks. However, their robustness under natural and adversarial perturbations remains a critical concern. We present a comprehensive evaluation of VSSMs' robustness under various perturbation scenarios.
  arXiv  Detail & Related papers  (2024-06-13T17:59:44Z)
• Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization [32.908816911260615]
  Vision transformers (ViTs) have been successfully deployed in a variety of computer vision tasks, but they are still vulnerable to adversarial samples. transfer-based attacks use a local model to generate adversarial samples and directly transfer them to attack a target black-box model. We propose the Token Gradient Regularization (TGR) method to overcome the shortcomings of existing approaches.
  arXiv  Detail & Related papers  (2023-03-28T06:23:17Z)
• Deeper Insights into ViTs Robustness towards Common Corruptions [82.79764218627558]
  We investigate how CNN-like architectural designs and CNN-based data augmentation strategies impact on ViTs' robustness towards common corruptions. We demonstrate that overlapping patch embedding and convolutional Feed-Forward Network (FFN) boost performance on robustness. We also introduce a novel conditional method enabling input-varied augmentations from two angles.
  arXiv  Detail & Related papers  (2022-04-26T08:22:34Z)
• From Environmental Sound Representation to Robustness of 2D CNN Models Against Adversarial Attacks [82.21746840893658]
  This paper investigates the impact of different standard environmental sound representations (spectrograms) on the recognition performance and adversarial attack robustness of a victim residual convolutional neural network. We show that while the ResNet-18 model trained on DWT spectrograms achieves a high recognition accuracy, attacking this model is relatively more costly for the adversary.
  arXiv  Detail & Related papers  (2022-04-14T15:14:08Z)
• On the Adversarial Robustness of Visual Transformers [129.29523847765952]
  This work provides the first and comprehensive study on the robustness of vision transformers (ViTs) against adversarial perturbations. Tested on various white-box and transfer attack settings, we find that ViTs possess better adversarial robustness when compared with convolutional neural networks (CNNs)
  arXiv  Detail & Related papers  (2021-03-29T14:48:24Z)
• Learn2Perturb: an End-to-end Feature Perturbation Learning to Improve Adversarial Robustness [79.47619798416194]
  Learn2Perturb is an end-to-end feature perturbation learning approach for improving the adversarial robustness of deep neural networks. Inspired by the Expectation-Maximization, an alternating back-propagation training algorithm is introduced to train the network and noise parameters consecutively.
  arXiv  Detail & Related papers  (2020-03-02T18:27:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.