PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers

• URL: http://arxiv.org/abs/2604.20047v1
• Date: Tue, 21 Apr 2026 23:04:49 GMT
• Title: PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers
• Abstract summary: We observe that a patch-wise trigger can achieve high attack effectiveness when activating backdoors across neighboring patches.<n>We propose PASTA, a twofold stealthy patch-wise backdoor attack in both pixel and attention domains.
• Score: 10.045003770844842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Transformers (ViTs) have achieved remarkable success across vision tasks, yet recent studies show they remain vulnerable to backdoor attacks. Existing patch-wise attacks typically assume a single fixed trigger location during inference to maximize trigger attention. However, they overlook the self-attention mechanism in ViTs, which captures long-range dependencies across patches. In this work, we observe that a patch-wise trigger can achieve high attack effectiveness when activating backdoors across neighboring patches, a phenomenon we term the Trigger Radiating Effect (TRE). We further find that inter-patch trigger insertion during training can synergistically enhance TRE compared to single-patch insertion. Prior ViT-specific attacks that maximize trigger attention often sacrifice visual and attention stealthiness, making them detectable. Based on these insights, we propose PASTA, a twofold stealthy patch-wise backdoor attack in both pixel and attention domains. PASTA enables backdoor activation when the trigger is placed at arbitrary patches during inference. To achieve this, we introduce a multi-location trigger insertion strategy to enhance TRE. However, preserving stealthiness while maintaining strong TRE is challenging, as TRE is weakened under stealthy constraints. We therefore formulate a bi-level optimization problem and propose an adaptive backdoor learning framework, where the model and trigger iteratively adapt to each other to avoid local optima. Extensive experiments show that PASTA achieves 99.13% attack success rate across arbitrary patches on average, while significantly improving visual and attention stealthiness (144.43x and 18.68x) and robustness (2.79x) against state-of-the-art ViT defenses across four datasets, outperforming CNN- and ViT-based baselines.

Related papers

• DETOUR: A Practical Backdoor Attack against Object Detection [10.045003770844842]
  We propose DETOUR, a practical backdoor attack by using semantic triggers that are effective in real-world object detection systems.<n>To ensure attack practicality, we rescale trigger patterns to different sizes and insert them at various predefined locations during backdoor training.<n>We extract the trigger pattern from a real-world object captured under multiple FoVs and inject the trigger accordingly, promoting viewpoint-invariant backdoor activation.
  arXiv  Detail & Related papers  (2026-04-27T15:25:55Z)
• Backdoor Directions in Vision Transformers [56.382912038371046]
  This paper investigates how Backdoor Attacks are represented within Vision Transformers (ViTs)<n>We identify a specific trigger direction'' in the model's activations that corresponds to the internal representation of the trigger.<n>Using this direction as a diagnostic tool, we trace how backdoor features are processed across layers.
  arXiv  Detail & Related papers  (2026-03-11T14:13:48Z)
• When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models [81.7618160628979]
  Vision-Language-Action (VLA) models are vulnerable to adversarial attacks, yet universal and transferable attacks remain underexplored.<n>We introduce UPA-RFAS (Universal Patch Attack via Robust Feature, Attention, and Semantics), a unified framework that learns a single physical patch in a shared feature space.<n> Experiments across diverse VLA models, manipulation suites, and physical executions show that UPA-RFAS consistently transfers across models, tasks, and viewpoints.
  arXiv  Detail & Related papers  (2025-11-26T09:16:32Z)
• Backdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning [5.0734761482919115]
  Open-vocabulary object detectors (OVODs) unify vision and language to detect arbitrary object categories based on text prompts.<n>We conduct the first study of backdoor attacks on OVODs and reveal a new attack surface introduced by prompt tuning.
  arXiv  Detail & Related papers  (2025-11-16T19:05:31Z)
• Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models [70.03122709795122]
  Backdoor attacks targeting text-to-image diffusion models have advanced rapidly.<n>Current backdoor samples often exhibit two key abnormalities compared to benign samples.<n>We propose Trigger without Trace (TwT) by explicitly mitigating these consistencies.
  arXiv  Detail & Related papers  (2025-03-22T10:41:46Z)
• Stealthy Patch-Wise Backdoor Attack in 3D Point Cloud via Curvature Awareness [52.780853311462636]
  Backdoor attacks pose a severe threat to deep neural networks (DNNs)<n>Existing 3D point cloud backdoor attacks rely on sample-wise global modifications.<n>We propose the Stealthy Patch-Wise Backdoor Attack (SPBA), the first patch-wise backdoor attack framework for 3D point clouds.
  arXiv  Detail & Related papers  (2025-03-12T12:30:59Z)
• Backdoor Attack Against Vision Transformers via Attention Gradient-Based Image Erosion [4.036142985883415]
  Vision Transformers (ViTs) have outperformed traditional Convolutional Neural Networks (CNN) across various computer vision tasks. ViTs are vulnerable to backdoor attacks, where an adversary embeds a backdoor into the victim model. We propose an Attention Gradient-based Erosion Backdoor (AGEB) targeted at ViTs.
  arXiv  Detail & Related papers  (2024-10-30T04:06:12Z)
• BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning [85.2564206440109]
  This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses. We introduce the emphtoolns attack, which is resistant to backdoor detection and model fine-tuning defenses.
  arXiv  Detail & Related papers  (2023-11-20T02:21:49Z)
• Defending Backdoor Attacks on Vision Transformer via Patch Processing [18.50522247164383]
  Vision Transformers (ViTs) have a radically different architecture with significantly less inductive bias than Convolutional Neural Networks. This paper investigates a representative causative attack, i.e., backdoor attacks. We propose an effective method for ViTs to defend both patch-based and blending-based trigger backdoor attacks via patch processing.
  arXiv  Detail & Related papers  (2022-06-24T17:29:47Z)
• Towards Transferable Adversarial Attacks on Vision Transformers [110.55845478440807]
  Vision transformers (ViTs) have demonstrated impressive performance on a series of computer vision tasks, yet they still suffer from adversarial examples. We introduce a dual attack framework, which contains a Pay No Attention (PNA) attack and a PatchOut attack, to improve the transferability of adversarial samples across different ViTs.
  arXiv  Detail & Related papers  (2021-09-09T11:28:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.