MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models

• URL: http://arxiv.org/abs/2502.08079v1
• Date: Wed, 12 Feb 2025 02:53:27 GMT
• Title: MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models
• Authors: Peng-Fei Zhang, Guangdong Bai, Zi Huang,
• Abstract summary: Meticulous Adrial Attack (MAA) fully exploit model-independent characteristics and vulnerabilities of individual samples. MAA emphasizes fine-grained optimization of adversarial images by developing a novel resizing and sliding crop (RScrop) technique.
• Score: 30.04163729936878
• License:
• Abstract: Current adversarial attacks for evaluating the robustness of vision-language pre-trained (VLP) models in multi-modal tasks suffer from limited transferability, where attacks crafted for a specific model often struggle to generalize effectively across different models, limiting their utility in assessing robustness more broadly. This is mainly attributed to the over-reliance on model-specific features and regions, particularly in the image modality. In this paper, we propose an elegant yet highly effective method termed Meticulous Adversarial Attack (MAA) to fully exploit model-independent characteristics and vulnerabilities of individual samples, achieving enhanced generalizability and reduced model dependence. MAA emphasizes fine-grained optimization of adversarial images by developing a novel resizing and sliding crop (RScrop) technique, incorporating a multi-granularity similarity disruption (MGSD) strategy. Extensive experiments across diverse VLP models, multiple benchmark datasets, and a variety of downstream tasks demonstrate that MAA significantly enhances the effectiveness and transferability of adversarial attacks. A large cohort of performance studies is conducted to generate insights into the effectiveness of various model configurations, guiding future advancements in this domain.

Related papers

• Modality Unified Attack for Omni-Modality Person Re-Identification [16.624135145315673]
  We propose a novel Modality Unified Attack method to train adversarial generators to attack different omni-modality models. Experiments show that our method can effectively attack the omni-modality re-id models, achieving 55.9%, 24.4%, 49.0% and 62.7% mean mAP Drop Rate.
  arXiv  Detail & Related papers  (2025-01-22T09:54:43Z)
• Towards Adversarially Robust Deep Metric Learning [0.8702432681310401]
  Deep neural networks are prone to adversarial attacks and could be easily fooled by adversarial examples. Existing works fail to thoroughly inspect the robustness of DML models. We propose a new defense, the Ensemble Adversarial Training (EAT), which exploits ensemble learning and adversarial training.
  arXiv  Detail & Related papers  (2025-01-02T03:15:25Z)
• Robust image classification with multi-modal large language models [4.709926629434273]
  adversarial examples can cause Deep Neural Networks to make incorrect predictions with high confidence. To mitigate these vulnerabilities, adversarial training and detection-based defenses have been proposed to strengthen models in advance. We propose a novel defense, Multi-Shield, designed to combine and complement these defenses with multi-modal information.
  arXiv  Detail & Related papers  (2024-12-13T18:49:25Z)
• Defensive Dual Masking for Robust Adversarial Defense [5.932787778915417]
  This paper introduces the Defensive Dual Masking (DDM) algorithm, a novel approach designed to enhance model robustness against such attacks. DDM utilizes a unique adversarial training strategy where [MASK] tokens are strategically inserted into training samples to prepare the model to handle adversarial perturbations more effectively. During inference, potentially adversarial tokens are dynamically replaced with [MASK] tokens to neutralize potential threats while preserving the core semantics of the input.
  arXiv  Detail & Related papers  (2024-12-10T00:41:25Z)
• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models [9.762046320216005]
  Large vision models have been found vulnerable to adversarial examples, emphasizing the need for enhancing their adversarial robustness. Recent approaches propose robust fine-tuning methods, such as adversarial tuning of low-rank adaptation (LoRA) in large vision models, but they still struggle to match the accuracy of full parameter adversarial fine-tuning. We propose hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously.
  arXiv  Detail & Related papers  (2024-10-08T12:05:01Z)
• Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality [69.76121008898677]
  Fine-grained Selective Calibrated CLIP integrates local hard negative loss and selective calibrated regularization. Our evaluations show that FSC-CLIP not only achieves compositionality on par with state-of-the-art models but also retains strong multi-modal capabilities.
  arXiv  Detail & Related papers  (2024-10-07T17:16:20Z)
• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models [55.73581212134293]
  We propose a novel, yet elegantly simple approach for detecting adversarial samples in Vision-Language Models. Our method leverages Text-to-Image (T2I) models to generate images based on captions produced by target VLMs. Empirical evaluations conducted on different datasets validate the efficacy of our approach.
  arXiv  Detail & Related papers  (2024-06-13T15:55:04Z)
• Model Stealing Attack against Graph Classification with Authenticity, Uncertainty and Diversity [80.16488817177182]
  GNNs are vulnerable to the model stealing attack, a nefarious endeavor geared towards duplicating the target model via query permissions. We introduce three model stealing attacks to adapt to different actual scenarios.
  arXiv  Detail & Related papers  (2023-12-18T05:42:31Z)
• SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation [56.622250514119294]
  In contrast to white-box adversarial attacks, transfer attacks are more reflective of real-world scenarios. We propose a self-augment-based transfer attack method, termed SA-Attack.
  arXiv  Detail & Related papers  (2023-12-08T09:08:50Z)
• Harnessing Perceptual Adversarial Patches for Crowd Counting [92.79051296850405]
  Crowd counting is vulnerable to adversarial examples in the physical world. This paper proposes the Perceptual Adrial Patch (PAP) generation framework to learn the shared perceptual features between models.
  arXiv  Detail & Related papers  (2021-09-16T13:51:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.