Stealthy and Adjustable Text-Guided Backdoor Attacks on Multimodal Pretrained Models

• URL: http://arxiv.org/abs/2604.05809v1
• Date: Tue, 07 Apr 2026 12:46:23 GMT
• Title: Stealthy and Adjustable Text-Guided Backdoor Attacks on Multimodal Pretrained Models
• Abstract summary: Multimodal pretrained models are vulnerable to backdoor attacks.<n>We propose a novel Text-Guided Backdoor (TGB) attack on multimodal pretrained models.
• Score: 65.63901005568772
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal pretrained models are vulnerable to backdoor attacks, yet most existing methods rely on visual or multimodal triggers, which are impractical since visually embedded triggers rarely occur in real-world data. To overcome this limitation, we propose a novel Text-Guided Backdoor (TGB) attack on multimodal pretrained models, where commonly occurring words in textual descriptions serve as backdoor triggers, significantly improving stealthiness and practicality. Furthermore, we introduce visual adversarial perturbations on poisoned samples to modulate the model's learning of textual triggers, enabling a controllable and adjustable TGB attack. Extensive experiments on downstream tasks built upon multimodal pretrained models, including Composed Image Retrieval (CIR) and Visual Question Answering (VQA), demonstrate that TGB achieves practicality and stealthiness with adjustable attack success rates across diverse realistic settings, revealing critical security vulnerabilities in multimodal pretrained models.

Related papers

• Backdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning [5.0734761482919115]
  Open-vocabulary object detectors (OVODs) unify vision and language to detect arbitrary object categories based on text prompts.<n>We conduct the first study of backdoor attacks on OVODs and reveal a new attack surface introduced by prompt tuning.
  arXiv  Detail & Related papers  (2025-11-16T19:05:31Z)
• Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning [89.1856483797116]
  We introduce BEAT, the first framework to inject visual backdoors into MLLM-based embodied agents.<n>Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably.<n>BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance.
  arXiv  Detail & Related papers  (2025-10-31T16:50:49Z)
• Robust Anti-Backdoor Instruction Tuning in LVLMs [53.766434746801366]
  We introduce a lightweight, certified-agnostic defense framework for large visual language models (LVLMs)<n>Our framework finetunes only adapter modules and text embedding layers under instruction tuning.<n>Experiments against seven attacks on Flickr30k and MSCOCO demonstrate that ours reduces their attack success rate to nearly zero.
  arXiv  Detail & Related papers  (2025-06-04T01:23:35Z)
• MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attacks [85.3303135160762]
  MIRAGE is a novel framework that exploits narrative-driven context and role immersion to circumvent safety mechanisms in Multimodal Large Language Models.<n>It achieves state-of-the-art performance, improving attack success rates by up to 17.5% over the best baselines.<n>We demonstrate that role immersion and structured semantic reconstruction can activate inherent model biases, facilitating the model's spontaneous violation of ethical safeguards.
  arXiv  Detail & Related papers  (2025-03-24T20:38:42Z)
• Robust image classification with multi-modal large language models [4.709926629434273]
  adversarial examples can cause Deep Neural Networks to make incorrect predictions with high confidence.<n>To mitigate these vulnerabilities, adversarial training and detection-based defenses have been proposed to strengthen models in advance.<n>We propose a novel defense, MultiShield, designed to combine and complement these defenses with multi-modal information.
  arXiv  Detail & Related papers  (2024-12-13T18:49:25Z)
• BadCM: Invisible Backdoor Attack Against Cross-Modal Learning [110.37205323355695]
  We introduce a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor. BadCM is the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework.
  arXiv  Detail & Related papers  (2024-10-03T03:51:53Z)
• VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models [65.23688155159398]
  Autoregressive Visual Language Models (VLMs) showcase impressive few-shot learning capabilities in a multimodal context. Recently, multimodal instruction tuning has been proposed to further enhance instruction-following abilities. Adversaries can implant a backdoor by injecting poisoned samples with triggers embedded in instructions or images. We propose a multimodal instruction backdoor attack, namely VL-Trojan.
  arXiv  Detail & Related papers  (2024-02-21T14:54:30Z)
• Adversarial Prompt Tuning for Vision-Language Models [86.5543597406173]
  Adversarial Prompt Tuning (AdvPT) is a technique to enhance the adversarial robustness of image encoders in Vision-Language Models (VLMs) We demonstrate that AdvPT improves resistance against white-box and black-box adversarial attacks and exhibits a synergistic effect when combined with existing image-processing-based defense techniques.
  arXiv  Detail & Related papers  (2023-11-19T07:47:43Z)
• VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models [46.14455492739906]
  Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting. We propose VLATTACK to generate adversarial samples by fusing perturbations of images and texts from both single-modal and multimodal levels.
  arXiv  Detail & Related papers  (2023-10-07T02:18:52Z)
• Exploring the Universal Vulnerability of Prompt-based Learning Paradigm [21.113683206722207]
  We find that prompt-based learning bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. We explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text.
  arXiv  Detail & Related papers  (2022-04-11T16:34:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.