Visual Prompting for Adversarial Robustness
- URL: http://arxiv.org/abs/2210.06284v4
- Date: Mon, 1 May 2023 02:22:39 GMT
- Title: Visual Prompting for Adversarial Robustness
- Authors: Aochuan Chen, Peter Lorenz, Yuguang Yao, Pin-Yu Chen, Sijia Liu
- Abstract summary: We use visual prompting computation to improve adversarial robustness of a fixed, pre-trained model at testing time.
We propose a new VP method, termed Class-wise Adrial Visual Prompting (C-AVP), to generate class-wise visual prompts.
C-AVP outperforms the conventional VP method, with 2.1X standard accuracy gain and 2X robust accuracy gain.
- Score: 63.89295305670113
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this work, we leverage visual prompting (VP) to improve adversarial
robustness of a fixed, pre-trained model at testing time. Compared to
conventional adversarial defenses, VP allows us to design universal (i.e.,
data-agnostic) input prompting templates, which have plug-and-play capabilities
at testing time to achieve desired model performance without introducing much
computation overhead. Although VP has been successfully applied to improving
model generalization, it remains elusive whether and how it can be used to
defend against adversarial attacks. We investigate this problem and show that
the vanilla VP approach is not effective in adversarial defense since a
universal input prompt lacks the capacity for robust learning against
sample-specific adversarial perturbations. To circumvent it, we propose a new
VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate
class-wise visual prompts so as to not only leverage the strengths of ensemble
prompts but also optimize their interrelations to improve model robustness. Our
experiments show that C-AVP outperforms the conventional VP method, with 2.1X
standard accuracy gain and 2X robust accuracy gain. Compared to classical
test-time defenses, C-AVP also yields a 42X inference time speedup.
Related papers
- TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models [53.91006249339802]
We propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks.
TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP.
We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets.
arXiv Detail & Related papers (2024-11-20T08:58:59Z) - Towards Robust Vision Transformer via Masked Adaptive Ensemble [23.986968861837813]
Adversarial training (AT) can help improve the robustness of Vision Transformers (ViT) against adversarial attacks.
This paper proposes a novel ViT architecture, including a detector and a classifier bridged by our newly developed adaptive ensemble.
Experimental results exhibit that our ViT architecture, on CIFAR-10, achieves the best standard accuracy and adversarial robustness of 90.3% and 49.8%, respectively.
arXiv Detail & Related papers (2024-07-22T05:28:29Z) - Improving Adversarial Robustness via Decoupled Visual Representation Masking [65.73203518658224]
In this paper, we highlight two novel properties of robust features from the feature distribution perspective.
We find that state-of-the-art defense methods aim to address both of these mentioned issues well.
Specifically, we propose a simple but effective defense based on decoupled visual representation masking.
arXiv Detail & Related papers (2024-06-16T13:29:41Z) - One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models [47.14654793461]
We present a Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC) to achieve the attack.
C-PGC incorporates both unimodal and cross-modal information as effective guidance.
Experiments show that C-PGC successfully forces adversarial samples to move away from their original area.
arXiv Detail & Related papers (2024-06-08T15:01:54Z) - Revisiting the Robust Generalization of Adversarial Prompt Tuning [4.033827046965844]
We propose an adaptive Consistency-guided Adrial Prompt Tuning (i.e., CAPT) framework to enhance the alignment of image and text features for adversarial examples.
We conduct experiments across 14 datasets and 4 data sparsity schemes to show the superiority of CAPT over other state-of-the-art adaption methods.
arXiv Detail & Related papers (2024-05-18T02:54:41Z) - Robust Feature Inference: A Test-time Defense Strategy using Spectral Projections [12.807619042576018]
We propose a novel test-time defense strategy called Robust Feature Inference (RFI)
RFI is easy to integrate with any existing (robust) training procedure without additional test-time computation.
We show that RFI improves robustness across adaptive and transfer attacks consistently.
arXiv Detail & Related papers (2023-07-21T16:18:58Z) - Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial
Robustness [53.094682754683255]
We propose a Model-Agnostic Meta-Attack (MAMA) approach to discover stronger attack algorithms automatically.
Our method learns the in adversarial attacks parameterized by a recurrent neural network.
We develop a model-agnostic training algorithm to improve the ability of the learned when attacking unseen defenses.
arXiv Detail & Related papers (2021-10-13T13:54:24Z) - Adaptive Feature Alignment for Adversarial Training [56.17654691470554]
CNNs are typically vulnerable to adversarial attacks, which pose a threat to security-sensitive applications.
We propose the adaptive feature alignment (AFA) to generate features of arbitrary attacking strengths.
Our method is trained to automatically align features of arbitrary attacking strength.
arXiv Detail & Related papers (2021-05-31T17:01:05Z) - Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness.
We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
arXiv Detail & Related papers (2020-10-26T04:44:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.