FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models

• URL: http://arxiv.org/abs/2509.06992v1
• Date: Wed, 03 Sep 2025 03:46:35 GMT
• Title: FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models
• Authors: Kun Zhai, Siheng Chen, Xingjun Ma, Yu-Gang Jiang,
• Abstract summary: Federated Adversarial Prompt Tuning (textbfFedAPT) is a novel method designed to enhance the adversarial robustness of FPT.<n>To address this issue, we propose a textbfclass-aware prompt generator that generates visual prompts from text prompts.<n>Experiments on multiple image classification datasets demonstrate the superiority of FedAPT in improving adversarial robustness.
• Score: 97.35577473867296
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated Prompt Tuning (FPT) is an efficient method for cross-client collaborative fine-tuning of large Vision-Language Models (VLMs). However, models tuned using FPT are vulnerable to adversarial attacks, leading to misclassification in downstream tasks. In this work, we introduce Federated Adversarial Prompt Tuning (\textbf{FedAPT}), a novel method designed to enhance the adversarial robustness of FPT. We identify a key issue in FedAPT under non-independent and identically distributed (non-IID) settings: a \textit{class information gap} between clients and the global model. Clients rely solely on limited local label information to generate adversarial samples for training, while the global model must defend against adversarial attacks from global labels. To address this issue, we propose a \textbf{class-aware prompt generator} that generates visual prompts from text prompts. This generator is guided by a \emph{Global Label Embedding} (serving as a ``beacon") which encodes cross-client label information to create more globally-aligned visual prompts. Additionally, we propose a \textbf{cross-layer generator sharing} strategy to enhance prompt coupling across different layers of the model, further boosting adversarial robustness. Extensive experiments on multiple image classification datasets demonstrate the superiority of FedAPT in improving adversarial robustness, outperforming existing methods by a large margin. FedAPT also exhibits exceptional generalization in cross-domain and cross-dataset scenarios, indicating its effectiveness in real-world applications.

Related papers

• FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learning [46.7918908485135]
  Federated Representation Entanglement (FedRE) is a framework built upon a novel form of client knowledge termed entangled representation.<n>FedRE achieves an effective trade-off among model performance, privacy protection, and communication overhead.
  arXiv  Detail & Related papers  (2025-11-27T09:42:22Z)
• Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers [5.231417382224748]
  We propose PEP-FedPT (Prompt Estimation from Prototypes for Federated Prompt Tuning) to achieve both generalization and personalization in visual prompt tuning of Vision Transformers (ViTs)<n>We introduce the novel Class-Contextualized Mixed Prompt (CCMP) - based on class-specific prompts maintained alongside a globally shared prompt.<n> PEP-FedPT consistently surpasses the state-of-the-art baselines under diverse data scenarios.
  arXiv  Detail & Related papers  (2025-10-29T10:42:56Z)
• Cooperative Pseudo Labeling for Unsupervised Federated Classification [62.9387841396335]
  Unsupervised Federated Learning (UFL) aims to collaboratively train a global model across distributed clients without sharing data or accessing label information.<n>We propose a novel method, underlinetextbfFederated underlinetextbfCooperative underlinetextbfPseudo underlinetextbfLabeling (textbfFedCoPL)<n>In particular, visual prompts containing general image features are aggregated at the server, while text prompts encoding personalized knowledge are retained locally.
  arXiv  Detail & Related papers  (2025-10-11T08:18:26Z)
• BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Models [8.874122783196139]
  We introduce textbfBadPromptFL, the first backdoor attack targeting prompt-based federated learning.<n>In BadPromptFL, compromised clients jointly optimize local backdoor triggers and prompt embeddings, injecting poisoned prompts into the global aggregation process.<n>Our experiments validate the effectiveness, stealth, and generalizability of our attack, raising critical concerns about the robustness of prompt-based federated learning.
  arXiv  Detail & Related papers  (2025-08-11T14:42:44Z)
• Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion [44.8670376715096]
  Federated learning (FL) enables collaborative model training across decentralized clients without sharing local data.<n>We propose a personalized FL framework based on dual-prompt learning and cross fusion, termed pFedDC.
  arXiv  Detail & Related papers  (2025-06-26T10:59:14Z)
• Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning [12.941603966989366]
  Federated Prototype Learning (FedPL) has emerged as an effective strategy for handling data heterogeneity in Federated Learning (FL)<n>We propose FedTSP, a novel method that leverages PLMs to construct semantically enriched prototypes from the textual modality.<n>To address the modality gap between client image models and the PLM, we introduce trainable prompts, allowing prototypes to adapt better to client tasks.
  arXiv  Detail & Related papers  (2025-03-16T04:35:06Z)
• Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks [64.02867484165476]
  To protect privacy and meet legal regulations, federated learning (FL) has gained significant attention for training speech-to-text (S2T) systems.<n>The commonly used FL approach (i.e., textscFedAvg) in S2T tasks typically suffers from extensive communication overhead.<n>We propose a personalized federated S2T framework that introduces textscFedLoRA, a lightweight LoRA module for client-side tuning and interaction with the server, and textscFedMem, a global model equipped with a $k$-near
  arXiv  Detail & Related papers  (2024-01-18T15:39:38Z)
• Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning [49.72857433721424]
  Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve state-of-the-art performance with improved efficiency in various computer vision tasks. We present a novel algorithm, SGPT, that integrates Generalized FL (GFL) and Personalized FL (PFL) approaches by employing a unique combination of both shared and group-specific prompts.
  arXiv  Detail & Related papers  (2023-10-27T17:22:09Z)
• Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
  Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. We find that the difference in logits between the local and global models increases as the model is continuously updated. We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
  arXiv  Detail & Related papers  (2023-08-20T04:41:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.