FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models

• URL: http://arxiv.org/abs/2510.20856v1
• Date: Wed, 22 Oct 2025 08:29:35 GMT
• Title: FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models
• Authors: Jia Deng, Jin Li, Zhenhua Zhao, Shaowei Wang,
• Abstract summary: We propose a new Test-Time defense: Feature Perception Threshold Counterattack Noise (FPT-Noise)<n>FPT-Noise enhances the adversarial robustness of CLIP without costly fine-tuning.<n>Extensive experimentation has demonstrated that FPT-Noise significantly outperforms existing Test-Time defense methods.
• Score: 22.747168689459468
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalizability across diverse downstream tasks. However, recent studies have revealed that VLMs, including CLIP, are highly vulnerable to adversarial attacks, particularly on their visual modality. Traditional methods for improving adversarial robustness, such as adversarial training, involve extensive retraining and can be computationally expensive. In this paper, we propose a new Test-Time defense: Feature Perception Threshold Counterattack Noise (FPT-Noise), which enhances the adversarial robustness of CLIP without costly fine-tuning. Our core contributions are threefold: First, we introduce a Dynamic Feature Modulator that dynamically generate an image-specific and attack-adaptive noise intensity parameter. Second, We reanalyzed the image features of CLIP. When images are exposed to different levels of noise, clean images and adversarial images exhibit distinct rates of feature change. We established a feature perception threshold to distinguish clean images from attacked ones. Finally, we integrate a Scene-Aware Regulation guided by a stability threshold and leverage Test-Time Transformation Ensembling (TTE) to further mitigate the impact of residual noise and enhance robustness.Extensive experimentation has demonstrated that FPT-Noise significantly outperforms existing Test-Time defense methods, boosting average robust accuracy from 0.07% to 56.86% under AutoAttack while maintaining high performance on clean images (-1.1%). The code will be made public following the publication of the study. The code will be made public following the publication of the study.

Related papers

• LeakBoost: Perceptual-Loss-Based Membership Inference Attack [4.82560917771631]
  LeakBoost is a perceptual-loss-based interrogation framework that actively probes a model's internal representations to expose hidden membership signals.<n>LeakBoost achieves substantial improvements at low false-positive rates across multiple image classification datasets and diverse neural network architectures.
  arXiv  Detail & Related papers  (2026-02-05T15:15:35Z)
• TTSnap: Test-Time Scaling of Diffusion Models via Noise-Aware Pruning [53.52543819839442]
  A prominent approach to test-time scaling for text-to-image diffusion models formulates the problem as a search over multiple noise seeds.<n>We propose test-time scaling with noise-aware pruning (TTSnap), a framework that prunes low-quality candidates without fully denoising them.
  arXiv  Detail & Related papers  (2025-11-27T09:14:26Z)
• Test-Time Defense Against Adversarial Attacks via Stochastic Resonance of Latent Ensembles [42.57676672281981]
  We propose a test-time defense mechanism against adversarial attacks.<n> imperceptible image perturbations that significantly alter the predictions of a model.<n>We show that our method recovers up to 68.1% of the accuracy loss on image classification, 71.9% on stereo matching, and 29.2% on optical flow under various types of adversarial attacks.
  arXiv  Detail & Related papers  (2025-10-03T17:57:25Z)
• CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP [54.660471826755234]
  We show that malicious perturbations that seek to maximise the classification loss lead to falsely stable' images.<n>We propose to leverage the pre-trained vision encoder of CLIP to counterattack such adversarial images during inference to achieve robustness.<n>Our paradigm is simple and training-free, providing the first method to defend CLIP from adversarial attacks at test time.
  arXiv  Detail & Related papers  (2025-03-05T15:51:59Z)
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models [53.91006249339802]
  We propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets.
  arXiv  Detail & Related papers  (2024-11-20T08:58:59Z)
• Confidence-aware Denoised Fine-tuning of Off-the-shelf Models for Certified Robustness [56.2479170374811]
  We introduce Fine-Tuning with Confidence-Aware Denoised Image Selection (FT-CADIS) FT-CADIS is inspired by the observation that the confidence of off-the-shelf classifiers can effectively identify hallucinated images during denoised smoothing. It has established the state-of-the-art certified robustness among denoised smoothing methods across all $ell$-adversary radius in various benchmarks.
  arXiv  Detail & Related papers  (2024-11-13T09:13:20Z)
• Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting [133.55037976429088]
  We investigate the adversarial robustness of vision transformers equipped with BERT pretraining (e.g., BEiT, MAE) A surprising observation is that MAE has significantly worse adversarial robustness than other BERT pretraining methods. We propose a simple yet effective way to boost the adversarial robustness of MAE.
  arXiv  Detail & Related papers  (2023-08-20T16:27:17Z)
• Beyond Pretrained Features: Noisy Image Modeling Provides Adversarial Defense [52.66971714830943]
  Masked image modeling (MIM) has made it a prevailing framework for self-supervised visual representation learning. In this paper, we investigate how this powerful self-supervised learning paradigm can provide adversarial robustness to downstream classifiers. We propose an adversarial defense method, referred to as De3, by exploiting the pretrained decoder for denoising.
  arXiv  Detail & Related papers  (2023-02-02T12:37:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.