BadCM: Invisible Backdoor Attack Against Cross-Modal Learning

• URL: http://arxiv.org/abs/2410.02182v1
• Date: Thu, 3 Oct 2024 03:51:53 GMT
• Title: BadCM: Invisible Backdoor Attack Against Cross-Modal Learning
• Authors: Zheng Zhang, Xu Yuan, Lei Zhu, Jingkuan Song, Liqiang Nie,
• Abstract summary: We introduce a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor. BadCM is the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework.
• Score: 110.37205323355695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite remarkable successes in unimodal learning tasks, backdoor attacks against cross-modal learning are still underexplored due to the limited generalization and inferior stealthiness when involving multiple modalities. Notably, since works in this area mainly inherit ideas from unimodal visual attacks, they struggle with dealing with diverse cross-modal attack circumstances and manipulating imperceptible trigger samples, which hinders their practicability in real-world applications. In this paper, we introduce a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor and propose a generalized invisible backdoor framework against cross-modal learning (BadCM). Specifically, a cross-modal mining scheme is developed to capture the modality-invariant components as target poisoning areas, where well-designed trigger patterns injected into these regions can be efficiently recognized by the victim models. This strategy is adapted to different image-text cross-modal models, making our framework available to various attack scenarios. Furthermore, for generating poisoned samples of high stealthiness, we conceive modality-specific generators for visual and linguistic modalities that facilitate hiding explicit trigger patterns in modality-invariant regions. To the best of our knowledge, BadCM is the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework. Comprehensive experimental evaluations on two typical applications, i.e., cross-modal retrieval and VQA, demonstrate the effectiveness and generalization of our method under multiple kinds of attack scenarios. Moreover, we show that BadCM can robustly evade existing backdoor defenses. Our code is available at https://github.com/xandery-geek/BadCM.

Related papers

• MakeupAttack: Feature Space Black-box Backdoor Attack on Face Recognition via Makeup Transfer [6.6251662169603005]
  We propose a novel feature backdoor attack against face recognition via makeup transfer, dubbed MakeupAttack. In our attack, we design an iterative training paradigm to learn the subtle features of the proposed makeup-style trigger. The results demonstrate that our proposed attack method can bypass existing state-of-the-art defenses while maintaining effectiveness, robustness, naturalness, and stealthiness, without compromising model performance.
  arXiv  Detail & Related papers  (2024-08-22T11:39:36Z)
• Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning [49.242828934501986]
  Multimodal contrastive learning has emerged as a powerful paradigm for building high-quality features. backdoor attacks subtly embed malicious behaviors within the model during training. We introduce an innovative token-based localized forgetting training regime.
  arXiv  Detail & Related papers  (2024-03-24T18:33:15Z)
• Shortcuts Everywhere and Nowhere: Exploring Multi-Trigger Backdoor Attacks [26.600846339400956]
  Backdoor attacks have become a significant threat to the pre-training and deployment of deep neural networks (DNNs) In this study, we explore the concept of Multi-Trigger Backdoor Attacks (MTBAs), where multiple adversaries leverage different types of triggers to poison the same dataset. By proposing and investigating three types of multi-trigger attacks including textitparallel, textitsequential, and textithybrid attacks, we demonstrate that 1) multiple triggers can coexist, overwrite, or cross-activate one another, and 2) MTBAs easily break the
  arXiv  Detail & Related papers  (2024-01-27T04:49:37Z)
• BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning [85.2564206440109]
  This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses. We introduce the emphtoolns attack, which is resistant to backdoor detection and model fine-tuning defenses.
  arXiv  Detail & Related papers  (2023-11-20T02:21:49Z)
• VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models [69.20464255450788]
  Diffusion Models (DMs) are state-of-the-art generative models that learn a reversible corruption process from iterative noise addition and denoising. Recent studies have shown that basic unconditional DMs are vulnerable to backdoor injection. This paper presents a unified backdoor attack framework to expand the current scope of backdoor analysis for DMs.
  arXiv  Detail & Related papers  (2023-06-12T05:14:13Z)
• Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models [23.695414399663235]
  This paper investigates the potential vulnerability of text-to-image (T2I) diffusion models to backdoor attacks via personalization. Our study focuses on a zero-day backdoor vulnerability prevalent in two families of personalization methods, epitomized by Textual Inversion and DreamBooth. By studying the prompt processing of Textual Inversion and DreamBooth, we have devised dedicated backdoor attacks according to the different ways of dealing with unseen tokens.
  arXiv  Detail & Related papers  (2023-05-18T04:28:47Z)
• Backdoor Attack with Sparse and Invisible Trigger [57.41876708712008]
  Deep neural networks (DNNs) are vulnerable to backdoor attacks. backdoor attack is an emerging yet threatening training-phase threat. We propose a sparse and invisible backdoor attack (SIBA)
  arXiv  Detail & Related papers  (2023-05-11T10:05:57Z)
• Untargeted Backdoor Attack against Object Detection [69.63097724439886]
  We design a poison-only backdoor attack in an untargeted manner, based on task characteristics. We show that, once the backdoor is embedded into the target model by our attack, it can trick the model to lose detection of any object stamped with our trigger patterns.
  arXiv  Detail & Related papers  (2022-11-02T17:05:45Z)
• Dual-Key Multimodal Backdoors for Visual Question Answering [26.988750557552983]
  We show that multimodal networks are vulnerable to a novel type of attack that we refer to as Dual-Key Multimodal Backdoors. This attack exploits the complex fusion mechanisms used by state-of-the-art networks to embed backdoors that are both effective and stealthy. We present an extensive study of multimodal backdoors on the Visual Question Answering (VQA) task with multiple architectures and visual feature backbones.
  arXiv  Detail & Related papers  (2021-12-14T18:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.