BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Models

• URL: http://arxiv.org/abs/2508.08040v3
• Date: Sat, 06 Sep 2025 08:54:45 GMT
• Title: BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Models
• Authors: Maozhen Zhang, Mengnan Zhao, Wei Wang, Bo Wang,
• Abstract summary: We introduce textbfBadPromptFL, the first backdoor attack targeting prompt-based federated learning.<n>In BadPromptFL, compromised clients jointly optimize local backdoor triggers and prompt embeddings, injecting poisoned prompts into the global aggregation process.<n>Our experiments validate the effectiveness, stealth, and generalizability of our attack, raising critical concerns about the robustness of prompt-based federated learning.
• Score: 8.874122783196139
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt-based tuning has emerged as a lightweight alternative to full fine-tuning in large vision-language models, enabling efficient adaptation via learned contextual prompts. This paradigm has recently been extended to federated learning settings (e.g., PromptFL), where clients collaboratively train prompts under data privacy constraints. However, the security implications of prompt-based aggregation in federated multimodal learning remain largely unexplored, leaving a critical attack surface unaddressed. In this paper, we introduce \textbf{BadPromptFL}, the first backdoor attack targeting prompt-based federated learning in multimodal contrastive models. In BadPromptFL, compromised clients jointly optimize local backdoor triggers and prompt embeddings, injecting poisoned prompts into the global aggregation process. These prompts are then propagated to benign clients, enabling universal backdoor activation at inference without modifying model parameters. Leveraging the contextual learning behavior of CLIP-style architectures, BadPromptFL achieves high attack success rates (e.g., \(>90\%\)) with minimal visibility and limited client participation. Extensive experiments across multiple datasets and aggregation protocols validate the effectiveness, stealth, and generalizability of our attack, raising critical concerns about the robustness of prompt-based federated learning in real-world deployments.

Related papers

• FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models [97.35577473867296]
  Federated Adversarial Prompt Tuning (textbfFedAPT) is a novel method designed to enhance the adversarial robustness of FPT.<n>To address this issue, we propose a textbfclass-aware prompt generator that generates visual prompts from text prompts.<n>Experiments on multiple image classification datasets demonstrate the superiority of FedAPT in improving adversarial robustness.
  arXiv  Detail & Related papers  (2025-09-03T03:46:35Z)
• BadFU: Backdoor Federated Learning through Adversarial Machine Unlearning [7.329446721934861]
  Federated learning (FL) has been widely adopted as a decentralized training paradigm.<n>In this paper, we present the first backdoor attack in the context of federated unlearning.
  arXiv  Detail & Related papers  (2025-08-21T13:17:01Z)
• SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learning [1.3312007032203859]
  We present the first study of backdoor attacks in Federated Prompt Learning.<n>We show that when malicious clients inject visually imperceptible, learnable noise triggers into input images, the global prompt learner becomes vulnerable to targeted misclassification.<n>Motivated by this vulnerability, we propose SABRE-FL, a lightweight, modular defense that filters poisoned prompt updates using an embedding-space anomaly detector trained offline on out-of-distribution data.
  arXiv  Detail & Related papers  (2025-06-25T23:15:20Z)
• R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning [69.72249695674665]
  We propose a robust test-time prompt tuning (R-TPT) for vision-language models (VLMs)<n>R-TPT mitigates the impact of adversarial attacks during the inference stage.<n>We introduce a plug-and-play reliability-based weighted ensembling strategy to strengthen the defense.
  arXiv  Detail & Related papers  (2025-04-15T13:49:31Z)
• Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models [7.810284483002312]
  Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning.<n>Current federated prompt learning methods are habitually restricted to the traditional FL paradigm, where the participating clients are generally only allowed to download a single globally aggregated model from the server.<n>We propose Personalized Federated Mixture of Adaptive Prompts (pFedMoAP), a novel FL framework that personalizes the prompt learning process through the lens of Mixture of Experts (MoE)
  arXiv  Detail & Related papers  (2024-10-14T03:05:12Z)
• Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning [49.72857433721424]
  Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve state-of-the-art performance with improved efficiency in various computer vision tasks. We present a novel algorithm, SGPT, that integrates Generalized FL (GFL) and Personalized FL (PFL) approaches by employing a unique combination of both shared and group-specific prompts.
  arXiv  Detail & Related papers  (2023-10-27T17:22:09Z)
• Inclusive Data Representation in Federated Learning: A Novel Approach Integrating Textual and Visual Prompt [12.869146009608816]
  We present Twin Prompt Federated learning (TPFL), a pioneering solution that integrates both visual and textual modalities. In order to tackle the data heterogeneity issues, we introduce the Augmented TPFL (ATPFL), which not only enhances the global knowledge acquisition of client models but also fosters the development of robust, compact models. The effectiveness of TPFL and ATPFL is substantiated by our extensive evaluations, consistently showing superior performance compared to all baselines.
  arXiv  Detail & Related papers  (2023-10-04T11:20:28Z)
• Backdoor Attacks in Peer-to-Peer Federated Learning [11.235386862864397]
  Peer-to-Peer Federated Learning (P2PFL) offer advantages in terms of both privacy and reliability. We propose new backdoor attacks for P2PFL that leverage structural graph properties to select the malicious nodes, and achieve high attack success.
  arXiv  Detail & Related papers  (2023-01-23T21:49:28Z)
• Bayesian Prompt Learning for Image-Language Model Generalization [64.50204877434878]
  We use the regularization ability of Bayesian methods to frame prompt learning as a variational inference problem. Our approach regularizes the prompt space, reduces overfitting to the seen prompts and improves the prompt generalization on unseen prompts. We demonstrate empirically on 15 benchmarks that Bayesian prompt learning provides an appropriate coverage of the prompt space.
  arXiv  Detail & Related papers  (2022-10-05T17:05:56Z)
• OpenPrompt: An Open-source Framework for Prompt-learning [59.17869696803559]
  We present OpenPrompt, a unified easy-to-use toolkit to conduct prompt-learning over PLMs. OpenPrompt is a research-friendly framework that is equipped with efficiency, modularity, and extendibility.
  arXiv  Detail & Related papers  (2021-11-03T03:31:14Z)
• RoFL: Attestable Robustness for Secure Federated Learning [59.63865074749391]
  Federated Learning allows a large number of clients to train a joint model without the need to share their private data. To ensure the confidentiality of the client updates, Federated Learning systems employ secure aggregation. We present RoFL, a secure Federated Learning system that improves robustness against malicious clients.
  arXiv  Detail & Related papers  (2021-07-07T15:42:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.