Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

• URL: http://arxiv.org/abs/2604.11576v1
• Date: Mon, 13 Apr 2026 14:54:25 GMT
• Title: Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models
• Abstract summary: We propose AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model.<n>Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss.<n>We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively.
• Score: 89.0460992131069
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite their impressive zero-shot abilities, vision-language models such as CLIP have been shown to be susceptible to adversarial attacks. To enhance its adversarial robustness, recent studies finetune the pretrained vision encoder of CLIP with adversarial examples on a proxy dataset such as ImageNet by aligning adversarial images with correct class labels. However, these methods overlook the important roles of training data distributions and learning objectives, resulting in reduced zero-shot capabilities and limited transferability of robustness across domains and datasets. In this work, we propose a simple yet effective paradigm AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model. Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss. To alleviate distortion of adversarial image embeddings of noisy web images, we further propose to regularise AdvFLYP by penalising deviation of adversarial image features. We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively. Extensive experiments on 14 downstream datasets spanning various domains show the superiority of our paradigm over mainstream practices. Our code and model weights are released at https://github.com/Sxing2/AdvFLYP.

Related papers

• Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment [65.51957843888061]
  Contrastive Language-Image Pre-training models are threatened by targeted data poisoning and attacks.<n>Previous defense methods correct poisoned image-caption pairs by matching a new caption for each image.<n>We propose an Optimal Transport-based framework to reconstruct image-caption pairs, named OTCCLIP.
  arXiv  Detail & Related papers  (2025-09-23T07:05:43Z)
• CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP [54.660471826755234]
  We show that malicious perturbations that seek to maximise the classification loss lead to falsely stable' images.<n>We propose to leverage the pre-trained vision encoder of CLIP to counterattack such adversarial images during inference to achieve robustness.<n>Our paradigm is simple and training-free, providing the first method to defend CLIP from adversarial attacks at test time.
  arXiv  Detail & Related papers  (2025-03-05T15:51:59Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks [52.26631767748843]
  We propose ROCLIP, the first effective method for robust pre-training multimodal vision-language models against targeted data poisoning and backdoor attacks. ROCLIP effectively breaks the association between poisoned image-caption pairs by considering a relatively large and varying pool of random captions. Our experiments show that ROCLIP renders state-of-the-art targeted data poisoning and backdoor attacks ineffective during pre-training CLIP models.
  arXiv  Detail & Related papers  (2023-03-13T04:49:46Z)
• Understanding Zero-Shot Adversarial Robustness for Large-Scale Models [31.295249927085475]
  We identify and explore the problem of emphadapting large-scale models for zero-shot adversarial robustness. We propose a text-guided contrastive adversarial training loss, which aligns the text embeddings and the adversarial visual features with contrastive learning. Our approach significantly improves the zero-shot adversarial robustness over CLIP, seeing an average improvement of over 31 points over ImageNet and 15 zero-shot datasets.
  arXiv  Detail & Related papers  (2022-12-14T04:08:56Z)
• Non-Contrastive Learning Meets Language-Image Pre-Training [145.6671909437841]
  We study the validity of non-contrastive language-image pre-training (nCLIP) We introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics.
  arXiv  Detail & Related papers  (2022-10-17T17:57:46Z)
• ProtoCLIP: Prototypical Contrastive Language Image Pretraining [12.067061175987075]
  Prototypical Contrastive Language Image Pretraining (ProtoCLIP) is introduced to enhance such grouping. ProtoCLIP sets up prototype-level discrimination between image and text spaces, which efficiently transfers higher-level structural knowledge. ProtoCLIP is trained with an online episodic training strategy, which makes it can be scaled up to unlimited amounts of data.
  arXiv  Detail & Related papers  (2022-06-22T11:55:53Z)
• Masked Unsupervised Self-training for Zero-shot Image Classification [98.23094305347709]
  Masked Unsupervised Self-Training (MUST) is a new approach which leverages two different and complimentary sources of supervision: pseudo-labels and raw images. MUST improves upon CLIP by a large margin and narrows the performance gap between unsupervised and supervised classification.
  arXiv  Detail & Related papers  (2022-06-07T02:03:06Z)
• Robust Cross-Modal Representation Learning with Progressive Self-Distillation [7.676408770854477]
  The learning objective of vision-language approach of CLIP does not effectively account for the noisy many-to-many correspondences found in web-harvested image captioning datasets. We introduce a novel training framework based on cross-modal contrastive learning that uses progressive self-distillation and soft image-text alignments to more efficiently learn robust representations from noisy data.
  arXiv  Detail & Related papers  (2022-04-10T03:28:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.