ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP

• URL: http://arxiv.org/abs/2511.17362v1
• Date: Fri, 21 Nov 2025 16:30:06 GMT
• Title: ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP
• Authors: Linxiang Su, András Balogh,
• Abstract summary: ATAC consistently achieves remarkably high robustness, surpassing that of previous state-of-the-art methods by nearly 50% on average.<n>ATAC is an efficient method in a novel paradigm for test-time adversarial defenses in the embedding space of CLIP.
• Score: 3.652509571098291
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite its remarkable success in zero-shot image-text matching, CLIP remains highly vulnerable to adversarial perturbations on images. As adversarial fine-tuning is prohibitively costly, recent works explore various test-time defense strategies; however, these approaches still exhibit limited robustness. In this work, we revisit this problem and propose a simple yet effective strategy: Augmentation-based Test-time Adversarial Correction (ATAC). Our method operates directly in the embedding space of CLIP, calculating augmentation-induced drift vectors to infer a semantic recovery direction and correcting the embedding based on the angular consistency of these latent drifts. Across a wide range of benchmarks, ATAC consistently achieves remarkably high robustness, surpassing that of previous state-of-the-art methods by nearly 50\% on average, all while requiring minimal computational overhead. Furthermore, ATAC retains state-of-the-art robustness in unconventional and extreme settings and even achieves nontrivial robustness against adaptive attacks. Our results demonstrate that ATAC is an efficient method in a novel paradigm for test-time adversarial defenses in the embedding space of CLIP.

Related papers

• Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP [68.44229678548298]
  Contrastive Spectral Rectification (CSR) is an efficient test-time defense against adversarial examples.<n>CSR outperforms the SOTA by an average of 18.1% against strong AutoAttack.<n>CSR exhibits broad applicability across diverse visual tasks.
  arXiv  Detail & Related papers  (2026-01-27T05:24:45Z)
• TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models [32.85951917559796]
  We propose Test-Time Padding (TTP), a lightweight defense framework that performs adversarial detection followed by targeted adaptation at inference.<n>TTP consistently surpasses state-of-the-art test-time defenses, delivering substantial improvements in adversarial robustness without compromising clean accuracy.
  arXiv  Detail & Related papers  (2025-12-18T13:34:14Z)
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models [53.91006249339802]
  We propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets.
  arXiv  Detail & Related papers  (2024-11-20T08:58:59Z)
• Adaptive Gradient Clipping for Robust Federated Learning [8.268485501864939]
  We propose a principled adaptive clipping strategy, Adaptive Robust Clipping (ARC), which dynamically adjusts clipping thresholds based on the input gradients.<n>ARC significantly enhances robustness, particularly in highly heterogeneous and adversarial settings.
  arXiv  Detail & Related papers  (2024-05-23T11:00:31Z)
• Revisiting and Advancing Adversarial Training Through A Simple Baseline [7.226961695849204]
  We introduce a simple baseline approach, termed SimpleAT, that performs competitively with recent methods and mitigates robust overfitting. We conduct extensive experiments on CIFAR-10/100 and Tiny-ImageNet, which validate the robustness of SimpleAT against state-of-the-art adversarial attackers. Our results also reveal the connections between SimpleAT and many advanced state-of-the-art adversarial defense methods.
  arXiv  Detail & Related papers  (2023-06-13T08:12:52Z)
• Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization [27.31806334022094]
  Recent research has shown an overfitting-like phenomenon in which models trained against adversarial attacks exhibit higher robustness on the training set compared to the test set. In this paper we consider a different form of the robust PAC-Bayesian bound and directly minimize it with respect to the model posterior. We evaluate our TrH regularization approach over CIFAR-10/100 and ImageNet using Vision Transformers (ViT) and compare against baseline adversarial robustness algorithms.
  arXiv  Detail & Related papers  (2022-11-22T23:12:00Z)
• Fast Adversarial Training with Adaptive Step Size [62.37203478589929]
  We study the phenomenon from the perspective of training instances. We propose a simple but effective method, Adversarial Training with Adaptive Step size (ATAS) ATAS learns an instancewise adaptive step size that is inversely proportional to its gradient norm.
  arXiv  Detail & Related papers  (2022-06-06T08:20:07Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)
• Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimization [60.72410937614299]
  We propose a new tractable bi-level optimization problem, design and analyze a new set of algorithms termed Bi-level AT (FAST-BAT) FAST-BAT is capable of defending sign-based projected descent (PGD) attacks without calling any gradient sign method and explicit robust regularization.
  arXiv  Detail & Related papers  (2021-12-23T06:25:36Z)
• ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients [65.52888259961803]
  We introduce ROPUST, a simple and efficient method to leverage robust pre-trained models and increase their robustness. We test our method on nine different models against four attacks in RobustBench, consistently improving over state-of-the-art performance. We show that even with state-of-the-art phase retrieval techniques, ROPUST remains an effective defense.
  arXiv  Detail & Related papers  (2021-07-06T12:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.