Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models

• URL: http://arxiv.org/abs/2606.06186v1
• Date: Thu, 04 Jun 2026 13:57:07 GMT
• Title: Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
• Abstract summary: Test-time defenses for vision-Language Models (VLMs) have emerged as a promising and efficient approach to defend against adversarial attacks without requiring large-scale retraining.<n>We propose Directional Bias-guided Defense (DBD), a test-time framework that estimates the Defense Direction and employs a DB-score-based two-stream reconstruction strategy to recover robust representations.<n>Experiments on 15 datasets demonstrate that DBD not only achieves SOTA adversarial robustness while preserving clean accuracy, but also reveals the counterintuitive result that adversarial accuracy can even surpass clean accuracy.
• Score: 44.54124778801062
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs), such as CLIP, have shown strong zero-shot generalization but remain highly vulnerable to adversarial perturbations, posing serious risks in real-world applications. Test-time defenses for VLMs have recently emerged as a promising and efficient approach to defend against adversarial attacks without requiring costly large-scale retraining. In this work, we uncover a surprising phenomenon: under diverse input transformations, adversarial images in CLIP's feature space consistently shift along a dominant direction, in contrast to the dispersed patterns of clean images. We hypothesize that this dominant shift, termed the Defense Direction, opposes the adversarial shift, pointing features back toward their correct class centers. Building on this insight, we propose Directional Bias-guided Defense (DBD), a test-time framework that estimates the Defense Direction and employs a DB-score-based two-stream reconstruction strategy to recover robust representations. Experiments on 15 datasets demonstrate that DBD not only achieves SOTA adversarial robustness while preserving clean accuracy, but also reveals the counterintuitive result that adversarial accuracy can even surpass clean accuracy. This demonstrates that adversarial perturbations inherently encode directional priors about the true decision boundary.

Related papers

• BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks [0.0]
  Supervised fine-tuning is the predominant approach for adapting autoregressive vision-language models to downstream tasks.<n>Recent work has shown that this paradigm is highly vulnerable to backdoor attacks.<n>We propose a backdoor-robust fine-tuning framework motivated by the observation that poisoned target responses are often semantically implausible.
  arXiv  Detail & Related papers  (2026-06-01T22:58:04Z)
• Zero-Sacrifice Persistent-Robustness Adversarial Defense for Pre-Trained Encoders [5.00483763729881]
  We introduce Zero-Sacrifice Persistent-Robustness Adversarial Defense (ZePAD)<n>ZePAD is inspired by the inherent sensitivity of neural networks to data characteristics.<n>It uses two adversarially fine-tuned encoders to strengthen adversarial resistance.
  arXiv  Detail & Related papers  (2026-02-10T07:41:34Z)
• Debiased Dual-Invariant Defense for Adversarially Robust Person Re-Identification [52.63017280231648]
  Person re-identification (ReID) is a fundamental task in many real-world applications such as pedestrian trajectory tracking.<n>Person ReID models are highly susceptible to adversarial attacks, where imperceptible perturbations to pedestrian images can cause entirely incorrect predictions.<n>We propose a dual-invariant defense framework composed of two main phases.
  arXiv  Detail & Related papers  (2025-11-13T03:56:40Z)
• Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference [45.723695657400576]
  We argue that enhancing the diversity and coverage of counterattacks is crucial to improving adversarial robustness in test-time defense.<n>We propose Directional Orthogonal Counterattack (DOC), which augments counterattack optimization by incorporating gradient directions and momentum-based updates.<n>We present a directional sensitivity score based on averaged cosine similarity to boost DOC by improving example discrimination and adaptively modulating the counterattack strength.
  arXiv  Detail & Related papers  (2025-11-12T07:40:16Z)
• GCP: Guarded Collaborative Perception with Spatial-Temporal Aware Malicious Agent Detection [11.336965062177722]
  Collaborative perception is vulnerable to adversarial message attacks from malicious agents.<n>This paper reveals a novel blind area confusion (BAC) attack that compromises existing single-shot outlier-based detection methods.<n>We propose Guarded Collaborative Perception framework based on spatial-temporal aware malicious agent detection.
  arXiv  Detail & Related papers  (2025-01-05T06:03:26Z)
• Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attacks [62.036798488144306]
  Current defense mainly focuses on the known attacks, but the adversarial robustness to the unknown attacks is seriously overlooked. We propose an attack-agnostic defense method named Meta Invariance Defense (MID) We show that MID simultaneously achieves robustness to the imperceptible adversarial perturbations in high-level image classification and attack-suppression in low-level robust image regeneration.
  arXiv  Detail & Related papers  (2024-04-04T10:10:38Z)
• Closing the Gap: Achieving Better Accuracy-Robustness Tradeoffs against Query-Based Attacks [1.54994260281059]
  We show how to efficiently establish, at test-time, a solid tradeoff between robustness and accuracy when mitigating query-based attacks. Our approach is independent of training and supported by theory.
  arXiv  Detail & Related papers  (2023-12-15T17:02:19Z)
• BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning [85.2564206440109]
  This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses. We introduce the emphtoolns attack, which is resistant to backdoor detection and model fine-tuning defenses.
  arXiv  Detail & Related papers  (2023-11-20T02:21:49Z)
• Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization [53.04697800214848]
  Adversarial Training (AT) is pivotal in fortifying the robustness of deep learning models. AT methods, relying on direct iterative updates for target model's defense, frequently encounter obstacles such as unstable training and catastrophic overfitting. We present a general proxy guided defense framework, LAST' (bf Learn from the Pbf ast)
  arXiv  Detail & Related papers  (2023-10-19T13:13:41Z)
• Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses [59.58128343334556]
  We introduce a relaxation term to the standard loss, that finds more suitable gradient-directions, increases attack efficacy and leads to more efficient adversarial training. We propose Guided Adversarial Margin Attack (GAMA), which utilizes function mapping of the clean image to guide the generation of adversaries. We also propose Guided Adversarial Training (GAT), which achieves state-of-the-art performance amongst single-step defenses.
  arXiv  Detail & Related papers  (2020-11-30T16:39:39Z)
• A Self-supervised Approach for Adversarial Robustness [105.88250594033053]
  Adversarial examples can cause catastrophic mistakes in Deep Neural Network (DNNs) based vision systems. This paper proposes a self-supervised adversarial training mechanism in the input space. It provides significant robustness against the textbfunseen adversarial attacks.
  arXiv  Detail & Related papers  (2020-06-08T20:42:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.