Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Models

• URL: http://arxiv.org/abs/2410.11639v2
• Date: Wed, 16 Oct 2024 13:48:37 GMT
• Title: Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Models
• Authors: Fan Yang, Yihao Huang, Kailong Wang, Ling Shi, Geguang Pu, Yang Liu, Haoyu Wang,
• Abstract summary: Non-universal adversarial attacks are often impractical for real-time online applications due to their high computational demands per data instance. We propose a direct optimization-based UAP approach, termed DO-UAP, which significantly reduces resource consumption while maintaining high attack performance.
• Score: 14.828324088905772
• License:
• Abstract: Vision-language pre-training (VLP) models, trained on large-scale image-text pairs, have become widely used across a variety of downstream vision-and-language (V+L) tasks. This widespread adoption raises concerns about their vulnerability to adversarial attacks. Non-universal adversarial attacks, while effective, are often impractical for real-time online applications due to their high computational demands per data instance. Recently, universal adversarial perturbations (UAPs) have been introduced as a solution, but existing generator-based UAP methods are significantly time-consuming. To overcome the limitation, we propose a direct optimization-based UAP approach, termed DO-UAP, which significantly reduces resource consumption while maintaining high attack performance. Specifically, we explore the necessity of multimodal loss design and introduce a useful data augmentation strategy. Extensive experiments conducted on three benchmark VLP datasets, six popular VLP models, and three classical downstream tasks demonstrate the efficiency and effectiveness of DO-UAP. Specifically, our approach drastically decreases the time consumption by 23-fold while achieving a better attack performance.

Related papers

• MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models [30.04163729936878]
  Meticulous Adrial Attack (MAA) fully exploit model-independent characteristics and vulnerabilities of individual samples. MAA emphasizes fine-grained optimization of adversarial images by developing a novel resizing and sliding crop (RScrop) technique.
  arXiv  Detail & Related papers  (2025-02-12T02:53:27Z)
• Efficient and Responsible Adaptation of Large Language Models for Robust and Equitable Top-k Recommendations [12.814937243361054]
  We propose a hybrid task allocation framework designed to promote social good by equitably serving all user groups. Our results on three real-world datasets show a significant reduction in weak users and improved robustness to subpopulations without disproportionately escalating costs.
  arXiv  Detail & Related papers  (2025-01-08T18:08:48Z)
• Adversarial Attacks on Large Language Models Using Regularized Relaxation [1.042748558542389]
  Large Language Models (LLMs) are used for numerous practical applications. adversarial attack methods are extensively used to study and understand these vulnerabilities. We propose a novel technique for adversarial attacks that overcomes these limitations by leveraging regularized gradients with continuous optimization methods.
  arXiv  Detail & Related papers  (2024-10-24T21:01:45Z)
• Efficient Adversarial Training in LLMs with Continuous Attacks [99.5882845458567]
  Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses. C-AdvIPO is an adversarial variant of IPO that does not require utility data for adversarially robust alignment.
  arXiv  Detail & Related papers  (2024-05-24T14:20:09Z)
• Universal Adversarial Perturbations for Vision-Language Pre-trained Models [30.04163729936878]
  We propose a novel black-box method to generate Universal Adversarial Perturbations (UAPs) The ETU takes into account the characteristics of UAPs and the intrinsic cross-modal interactions to generate effective UAPs. To further enhance the effectiveness and transferability of UAPs, we also design a novel data augmentation method named ScMix.
  arXiv  Detail & Related papers  (2024-05-09T03:27:28Z)
• Effective Backdoor Mitigation in Vision-Language Models Depends on the Pre-training Objective [71.39995120597999]
  Modern machine learning models are vulnerable to adversarial and backdoor attacks. Such risks are heightened by the prevalent practice of collecting massive, internet-sourced datasets for training multimodal models. CleanCLIP is the current state-of-the-art approach to mitigate the effects of backdooring in multimodal models.
  arXiv  Detail & Related papers  (2023-11-25T06:55:13Z)
• Avoid Adversarial Adaption in Federated Learning by Multi-Metric Investigations [55.2480439325792]
  Federated Learning (FL) facilitates decentralized machine learning model training, preserving data privacy, lowering communication costs, and boosting model performance through diversified data sources. FL faces vulnerabilities such as poisoning attacks, undermining model integrity with both untargeted performance degradation and targeted backdoor attacks. We define a new notion of strong adaptive adversaries, capable of adapting to multiple objectives simultaneously. MESAS is the first defense robust against strong adaptive adversaries, effective in real-world data scenarios, with an average overhead of just 24.37 seconds.
  arXiv  Detail & Related papers  (2023-06-06T11:44:42Z)
• On Evaluating Adversarial Robustness of Large Vision-Language Models [64.66104342002882]
  We evaluate the robustness of large vision-language models (VLMs) in the most realistic and high-risk setting. In particular, we first craft targeted adversarial examples against pretrained models such as CLIP and BLIP. Black-box queries on these VLMs can further improve the effectiveness of targeted evasion.
  arXiv  Detail & Related papers  (2023-05-26T13:49:44Z)
• Dynamic Transformers Provide a False Sense of Efficiency [75.39702559746533]
  Multi-exit models make a trade-off between efficiency and accuracy, where the saving of computation comes from an early exit. We propose a simple yet effective attacking framework, SAME, which is specially tailored to reduce the efficiency of the multi-exit models. Experiments on the GLUE benchmark show that SAME can effectively diminish the efficiency gain of various multi-exit models by 80% on average.
  arXiv  Detail & Related papers  (2023-05-20T16:41:48Z)
• Cluster-level pseudo-labelling for source-free cross-domain facial expression recognition [94.56304526014875]
  We propose the first Source-Free Unsupervised Domain Adaptation (SFUDA) method for Facial Expression Recognition (FER) Our method exploits self-supervised pretraining to learn good feature representations from the target data. We validate the effectiveness of our method in four adaptation setups, proving that it consistently outperforms existing SFUDA methods when applied to FER.
  arXiv  Detail & Related papers  (2022-10-11T08:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.