Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models

• URL: http://arxiv.org/abs/2606.03730v2
• Date: Thu, 04 Jun 2026 14:10:51 GMT
• Title: Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
• Abstract summary: Vision-language models (VLMs) such as CLIP show strong zero-shot generalization but remain highly vulnerable to adversarial attacks.<n>Recent approaches exploit how CLIP's visual representations respond to perturbations.<n>We propose a training-free, plug-in drift-gated mechanism that uses high-noise feature drift as a lightweight gating signal.
• Score: 37.24612885977265
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) such as CLIP show strong zero-shot generalization but remain highly vulnerable to adversarial attacks. Adversarial training improves robustness but is computationally expensive, motivating test-time defenses. Recent approaches exploit how CLIP's visual representations respond to stochastic perturbations: aggregating predictions across noisy views, constructing Gaussian noise-averaged anchors and interpolating features toward them, or applying counter-perturbations. These strategies improve robustness but often degrade clean accuracy, yielding an unfavorable clean-robust trade-off. We revisit stochastic test-time defenses and identify an underexplored noise-regime transition in CLIP's representation space. Prior work explored perturbations mainly in the weak-noise regime, where adversarial examples can appear unusually stable (false stability). Our analysis shows this reverses as perturbation strength grows: beyond the weak-noise regime, adversarial representations become markedly more unstable than clean ones, giving a clearer separation signal. The transition is consistent across uniform and Gaussian noise, photometric and geometric transforms, datasets, and diverse attacks. It largely disappears in adversarially trained models, suggesting it is tied to the fragile local-basin geometry of adversarial representations in non-robust CLIP. We propose a training-free, plug-in drift-gated mechanism that uses high-noise feature drift as a lightweight gating signal to trigger existing test-time defenses only when adversarial-like instability is detected. Across 13 datasets it consistently improves the clean-robust trade-off. On eight fine-grained datasets, mean clean+adversarial accuracy rises from 65.7% to 71.4% for counterattack defenses and 68.4% to 73.2% for noise-anchoring; on ImageNet and four shifted variants, from 56.1% to 66.2% and 62.1% to 67.6%.

Related papers

• Delay-Aware Active Triangulation with Uncertainty-Driven Multi-Agent Reinforcement Learning for Counter-UAS [5.4271995156436965]
  Multi-agent active visual triangulation enables precise 3D localization of aerial targets by coordinating mobile observers with controllable cameras.<n>We present a delay-aware, uncertainty-driven multi-agent reinforcement learning framework for target localization in Counter-UAS applications.
  arXiv  Detail & Related papers  (2026-07-07T07:58:13Z)
• Multi-Axis Trust Modeling for Interpretable Account Hijacking Detection [1.0152838128195467]
  This paper proposes a Hadith-inspired multi-axis trust modeling framework.<n>We translate five trust axes - long-term integrity (adalah), behavioral precision (dabt), contextual continuity (isnad), cumulative reputation, and anomaly evidence - into a compact set of 26 semantically meaningful behavioral features for user accounts.<n>We evaluate the framework on the CLUE-LDS cloud activity dataset with injected account hijacking scenarios.
  arXiv  Detail & Related papers  (2026-02-20T19:36:30Z)
• BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning [73.46118996284888]
  Research on backdoor attacks against multimodal contrastive learning models faces two key challenges: stealthiness and persistence.<n>We propose BadCLIP++, a unified framework that tackles both challenges.<n>For stealthiness, we introduce a semantic-fusion QR micro-trigger that embeds imperceptible patterns near task-relevant regions.<n>For persistence, we stabilize trigger embeddings via radius shrinkage and centroid alignment.
  arXiv  Detail & Related papers  (2026-02-19T08:31:16Z)
• Test-Time Defense Against Adversarial Attacks via Stochastic Resonance of Latent Ensembles [42.57676672281981]
  We propose a test-time defense mechanism against adversarial attacks.<n> imperceptible image perturbations that significantly alter the predictions of a model.<n>We show that our method recovers up to 68.1% of the accuracy loss on image classification, 71.9% on stereo matching, and 29.2% on optical flow under various types of adversarial attacks.
  arXiv  Detail & Related papers  (2025-10-03T17:57:25Z)
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models [53.91006249339802]
  We propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets.
  arXiv  Detail & Related papers  (2024-11-20T08:58:59Z)
• MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsification [34.49704185920974]
  MeanSparse is a method to improve the robustness of Convolutional and attention-based Neural Networks against adversarial examples.<n>Our experiments show that MeanSparse achieves a new robustness record of 75.28%.
  arXiv  Detail & Related papers  (2024-06-09T22:14:55Z)
• Certified Robustness Against Natural Language Attacks by Causal Intervention [61.62348826831147]
  Causal Intervention by Semantic Smoothing (CISS) is a novel framework towards robustness against natural language attacks. CISS is provably robust against word substitution attacks, as well as empirically robust even when perturbations are strengthened by unknown attack algorithms.
  arXiv  Detail & Related papers  (2022-05-24T19:20:48Z)
• From Environmental Sound Representation to Robustness of 2D CNN Models Against Adversarial Attacks [82.21746840893658]
  This paper investigates the impact of different standard environmental sound representations (spectrograms) on the recognition performance and adversarial attack robustness of a victim residual convolutional neural network. We show that while the ResNet-18 model trained on DWT spectrograms achieves a high recognition accuracy, attacking this model is relatively more costly for the adversary.
  arXiv  Detail & Related papers  (2022-04-14T15:14:08Z)
• Towards Adversarial Patch Analysis and Certified Defense against Crowd Counting [61.99564267735242]
  Crowd counting has drawn much attention due to its importance in safety-critical surveillance systems. Recent studies have demonstrated that deep neural network (DNN) methods are vulnerable to adversarial attacks. We propose a robust attack strategy called Adversarial Patch Attack with Momentum to evaluate the robustness of crowd counting models.
  arXiv  Detail & Related papers  (2021-04-22T05:10:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.