FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts

• URL: http://arxiv.org/abs/2511.00480v1
• Date: Sat, 01 Nov 2025 10:15:04 GMT
• Title: FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
• Authors: Weihao Bo, Yanpeng Sun, Yu Wang, Xinyu Zhang, Zechao Li,
• Abstract summary: FedMGP is a new paradigm for personalized federated prompt learning in vision-language models.<n>A diversity loss is introduced to drive each prompt group to specialize in distinct and complementary semantic aspects.<n>FedMGP consistently outperforms prior approaches in both personalization and domain generalization.
• Score: 31.907894865146385
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we introduce FedMGP, a new paradigm for personalized federated prompt learning in vision-language models. FedMGP equips each client with multiple groups of paired textual and visual prompts, enabling the model to capture diverse, fine-grained semantic and instance-level cues. A diversity loss is introduced to drive each prompt group to specialize in distinct and complementary semantic aspects, ensuring that the groups collectively cover a broader range of local characteristics. During communication, FedMGP employs a dynamic prompt aggregation strategy based on similarity-guided probabilistic sampling: each client computes the cosine similarity between its prompt groups and the global prompts from the previous round, then samples s groups via a softmax-weighted distribution. This soft selection mechanism preferentially aggregates semantically aligned knowledge while still enabling exploration of underrepresented patterns effectively balancing the preservation of common knowledge with client-specific features. Notably, FedMGP maintains parameter efficiency by redistributing a fixed prompt capacity across multiple groups, achieving state-of-the-art performance with the lowest communication parameters among all federated prompt learning methods. Theoretical analysis shows that our dynamic aggregation strategy promotes robust global representation learning by reinforcing shared semantics while suppressing client-specific noise. Extensive experiments demonstrate that FedMGP consistently outperforms prior approaches in both personalization and domain generalization across diverse federated vision-language benchmarks. The code will be released on https://github.com/weihao-bo/FedMGP.git.

Related papers

• Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models [67.45032003041399]
  We propose a novel Multi-Paradigm Collaborative Attack (MPCAttack) framework to boost the transferability of adversarial examples against MLLMs.<n>MPCO adaptively balances the importance of different paradigm representations and guides the global optimisation.<n>Our solution consistently outperforms state-of-the-art methods in both targeted and untargeted attacks on open-source and closed-source MLLMs.
  arXiv  Detail & Related papers  (2026-03-05T06:01:26Z)
• Federated Multi-Task Clustering [44.73672172790804]
  This paper proposes a novel framework named Federated Multi-Task Clustering (i.e.,FMTC)<n>It is composed of two main components: client-side personalized clustering module and server-side tensorial correlation module.<n>We derive an efficient, privacy-preserving distributed algorithm based on the Alternating Direction Method of Multipliers.
  arXiv  Detail & Related papers  (2025-12-28T12:02:32Z)
• Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data [24.083180898232417]
  We propose a new federated learning framework featuring locally adaptive representations based on learnable client-side embedding controls.<n>These embeddings serve as reconfiguration signals that align the globally aggregated representation with each client's local context.<n>The proposed method achieves up to 36.45% performance improvement under severe data incompleteness.
  arXiv  Detail & Related papers  (2025-10-27T00:09:58Z)
• FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models [97.35577473867296]
  Federated Adversarial Prompt Tuning (textbfFedAPT) is a novel method designed to enhance the adversarial robustness of FPT.<n>To address this issue, we propose a textbfclass-aware prompt generator that generates visual prompts from text prompts.<n>Experiments on multiple image classification datasets demonstrate the superiority of FedAPT in improving adversarial robustness.
  arXiv  Detail & Related papers  (2025-09-03T03:46:35Z)
• Federated Cross-Modal Style-Aware Prompt Generation [2.4472081831862655]
  FedCSAP produces context-aware prompt tokens that are both distinct and non-redundant.<n>Our framework harnesses low, mid, and high-level features from CLIP's vision encoder alongside client-specific style indicators.
  arXiv  Detail & Related papers  (2025-08-17T15:23:45Z)
• Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization [51.562474873972086]
  Federated domain generalization (FedDG) aims to learn a globally generalizable model from decentralized clients with heterogeneous data.<n>Recent studies have introduced prompt learning to adapt vision-language models (VLMs) in FedDG by learning a single global prompt.<n>We propose TRIP, a Token-level prompt mixture with parameter-free routing framework for FedDG.
  arXiv  Detail & Related papers  (2025-04-29T11:06:03Z)
• Byzantine Resilient Federated Multi-Task Representation Learning [8.501036982144454]
  We propose BR-MTRL, a Byzantine-resilient multi-task representation learning framework that handles faulty or malicious agents.<n>Our approach leverages representation learning through a shared neural network model, where all clients share fixed layers, except for a client-specific final layer.
  arXiv  Detail & Related papers  (2025-03-24T23:26:28Z)
• Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances [24.142013877384603]
  This paper introduces a novel unsupervised multimodal clustering method (UMC), making a pioneering contribution to this field. UMC introduces a unique approach to constructing augmentation views for multimodal data, which are then used to perform pre-training. We show remarkable improvements of 2-6% scores in clustering metrics over state-of-the-art methods, marking the first successful endeavor in this domain.
  arXiv  Detail & Related papers  (2024-05-21T13:24:07Z)
• Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning [49.72857433721424]
  Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve state-of-the-art performance with improved efficiency in various computer vision tasks. We present a novel algorithm, SGPT, that integrates Generalized FL (GFL) and Personalized FL (PFL) approaches by employing a unique combination of both shared and group-specific prompts.
  arXiv  Detail & Related papers  (2023-10-27T17:22:09Z)
• Personalized Federated Learning via Amortized Bayesian Meta-Learning [21.126405589760367]
  We introduce a new perspective on personalized federated learning through Amortized Bayesian Meta-Learning. Specifically, we propose a novel algorithm called emphFedABML, which employs hierarchical variational inference across clients. Our theoretical analysis provides an upper bound on the average generalization error and guarantees the generalization performance on unseen data.
  arXiv  Detail & Related papers  (2023-07-05T11:58:58Z)
• Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach [66.9033666087719]
  This paper extends the inference view and describes a variational inference formulation of federated learning. We apply FedEP on standard federated learning benchmarks and find that it outperforms strong baselines in terms of both convergence speed and accuracy.
  arXiv  Detail & Related papers  (2023-02-08T17:58:11Z)
• On the Convergence of Clustered Federated Learning [57.934295064030636]
  In a federated learning system, the clients, e.g. mobile devices and organization participants, usually have different personal preferences or behavior patterns. This paper proposes a novel weighted client-based clustered FL algorithm to leverage the client's group and each client in a unified optimization framework.
  arXiv  Detail & Related papers  (2022-02-13T02:39:19Z)
• Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
  We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
  arXiv  Detail & Related papers  (2021-02-14T05:36:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.