Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness

• URL: http://arxiv.org/abs/2603.18598v1
• Date: Thu, 19 Mar 2026 08:11:46 GMT
• Title: Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness
• Abstract summary: We propose a Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR) framework.<n>Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness.<n>To overcome this limitation, we propose a novel approach called Complementary Text-Guided Attention (Comp-TGA)
• Score: 57.104158692005775
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Due to the impressive zero-shot capabilities, pre-trained vision-language models (e.g., CLIP), have attracted widespread attention and adoption across various domains. Nonetheless, CLIP has been observed to be susceptible to adversarial examples. Through experimental analysis, we have observed a phenomenon wherein adversarial perturbations induce shifts in text-guided attention. Building upon this observation, we propose a simple yet effective strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This framework incorporates two components: Local Attention Refinement Module and Global Attention Constraint Module. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness. Additionally, the Global Attention Constraint Module acquires text-guided attention from both the target and original models using clean examples. Its objective is to maintain model performance on clean samples while enhancing overall robustness. However, we observe that the method occasionally focuses on irrelevant or spurious features, which can lead to suboptimal performance and undermine its robustness in certain scenarios. To overcome this limitation, we further propose a novel approach called Complementary Text-Guided Attention (Comp-TGA). This method integrates two types of foreground attention: attention guided by the class prompt and reversed attention driven by the non-class prompt. These complementary attention mechanisms allow the model to capture a more comprehensive and accurate representation of the foreground. The experiments validate that TGA-ZSR and Comp-TGA yield 9.58% and 11.95% improvements respectively, in zero-shot robust accuracy over the current state-of-the-art techniques across 16 datasets.

Related papers

• DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection [16.74080857973598]
  We propose a fully automated anomaly detection framework leveraging global context to remove manual tunings.<n>Our method achieves Image-AUC scores of 97.7%, 93.2%, and 84.5% on MVTec-AD, VisA, and Real-IAD datasets.
  arXiv  Detail & Related papers  (2026-07-27T01:31:48Z)
• AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models [12.774216017720642]
  Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations.<n>We propose an Alignment-Guided Fine-Tuning (AGFT) framework that enhances zero-shot adversarial robustness while preserving the cross-modal semantic structure.
  arXiv  Detail & Related papers  (2026-03-31T08:17:27Z)
• Zero-Shot Robustness of Vision Language Models Via Confidence-Aware Weighting [1.5268922363885407]
  We propose Confidence-Aware Weighting (CAW) to enhance zero-shot robustness in vision-language models.<n>CAW consists of two components: (1) a Confidence-Aware loss that prioritizes uncertain adversarial examples by scaling the KL divergence between clean and adversarial predictions, and (2) a feature alignment regularization that preserves semantic consistency.
  arXiv  Detail & Related papers  (2025-10-03T11:36:02Z)
• Point Cloud Understanding via Attention-Driven Contrastive Learning [64.65145700121442]
  Transformer-based models have advanced point cloud understanding by leveraging self-attention mechanisms. PointACL is an attention-driven contrastive learning framework designed to address these limitations. Our method employs an attention-driven dynamic masking strategy that guides the model to focus on under-attended regions.
  arXiv  Detail & Related papers  (2024-11-22T05:41:00Z)
• Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models [64.67721492968941]
  We propose a Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR) framework. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness. Our method yields a 9.58% enhancement in zero-shot robust accuracy over the current state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-10-29T07:15:09Z)
• Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement [68.31147013783387]
  We observe that the attention mechanism is vulnerable to patch-based adversarial attacks. In this paper, we propose a Robust Attention Mechanism (RAM) to improve the robustness of the semantic segmentation model.
  arXiv  Detail & Related papers  (2024-01-03T13:58:35Z)
• Understanding Zero-Shot Adversarial Robustness for Large-Scale Models [31.295249927085475]
  We identify and explore the problem of emphadapting large-scale models for zero-shot adversarial robustness. We propose a text-guided contrastive adversarial training loss, which aligns the text embeddings and the adversarial visual features with contrastive learning. Our approach significantly improves the zero-shot adversarial robustness over CLIP, seeing an average improvement of over 31 points over ImageNet and 15 zero-shot datasets.
  arXiv  Detail & Related papers  (2022-12-14T04:08:56Z)
• When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning? [99.4914671654374]
  We propose AdvCL, a novel adversarial contrastive pretraining framework. We show that AdvCL is able to enhance cross-task robustness transferability without loss of model accuracy and finetuning efficiency.
  arXiv  Detail & Related papers  (2021-11-01T17:59:43Z)
• More Than Just Attention: Learning Cross-Modal Attentions with Contrastive Constraints [63.08768589044052]
  We propose Contrastive Content Re-sourcing ( CCR) and Contrastive Content Swapping ( CCS) constraints to address such limitation. CCR and CCS constraints supervise the training of attention models in a contrastive learning manner without requiring explicit attention annotations. Experiments on both Flickr30k and MS-COCO datasets demonstrate that integrating these attention constraints into two state-of-the-art attention-based models improves the model performance.
  arXiv  Detail & Related papers  (2021-05-20T08:48:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.