PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model

• URL: http://arxiv.org/abs/2208.11625v1
• Date: Wed, 24 Aug 2022 15:50:58 GMT
• Title: PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model
• Authors: Tao Guo, Song Guo, Junxiao Wang, Wenchao Xu
• Abstract summary: We propose a brand-new FL framework, PromptFL, that replaces the federated model training with the federated prompt training. PromptFL ships an off-the-shelf FM, i.e., CLIP, to distributed clients who would cooperatively train shared soft prompts. We empirically analyze the PromptFL via extensive experiments, and show its superiority in terms of system feasibility, user privacy, and performance.
• Score: 23.916918530195826
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quick global aggregation of effective distributed parameters is crucial to federated learning (FL), which requires adequate bandwidth for parameters communication and sufficient user data for local training. Otherwise, FL may cost excessive training time for convergence and produce inaccurate models. In this paper, we propose a brand-new FL framework, PromptFL, that replaces the federated model training with the federated prompt training, i.e., let federated participants train prompts instead of a shared model, to simultaneously achieve the efficient global aggregation and local training on insufficient data by exploiting the power of foundation models (FM) in a distributed way. PromptFL ships an off-the-shelf FM, i.e., CLIP, to distributed clients who would cooperatively train shared soft prompts based on very few local data. Since PromptFL only needs to update the prompts instead of the whole model, both the local training and the global aggregation can be significantly accelerated. And FM trained over large scale data can provide strong adaptation capability to distributed users tasks with the trained soft prompts. We empirically analyze the PromptFL via extensive experiments, and show its superiority in terms of system feasibility, user privacy, and performance.

Related papers

• CELLM: An Efficient Communication in Large Language Models Training for Federated Learning [0.0]
  This thesis aims to develop efficient training methods for large language models (LLMs) in Federated Learning (FL) First, we use low-rank adaptation (LoRA) to reduce the computational load of local model training. Second, we communicate sparse updates throughout training to significantly cut down on communication costs.
  arXiv  Detail & Related papers  (2024-07-30T05:24:08Z)
• Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training [21.89214794178211]
  In Federated Learning (FL), clients may have weak devices that cannot train the full model or even hold it in their memory space. We propose EmbracingFL, a general FL framework that allows all available clients to join the distributed training. Our empirical study shows that EmbracingFL consistently achieves high accuracy as like all clients are strong, outperforming the state-of-the-art width reduction methods.
  arXiv  Detail & Related papers  (2024-06-21T13:19:29Z)
• A Survey on Efficient Federated Learning Methods for Foundation Model Training [62.473245910234304]
  Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
  arXiv  Detail & Related papers  (2024-01-09T10:22:23Z)
• Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
  Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources. The lack of model privacy protection in FL becomes an unneglectable challenge. We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
  arXiv  Detail & Related papers  (2023-11-12T11:01:10Z)
• Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning [49.72857433721424]
  Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve state-of-the-art performance with improved efficiency in various computer vision tasks. We present a novel algorithm, SGPT, that integrates Generalized FL (GFL) and Personalized FL (PFL) approaches by employing a unique combination of both shared and group-specific prompts.
  arXiv  Detail & Related papers  (2023-10-27T17:22:09Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• Achieving Personalized Federated Learning with Sparse Local Models [75.76854544460981]
  Federated learning (FL) is vulnerable to heterogeneously distributed data. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. Existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. We proposeFedSpa, a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.
  arXiv  Detail & Related papers  (2022-01-27T08:43:11Z)
• Federated Robustness Propagation: Sharing Adversarial Robustness in Federated Learning [98.05061014090913]
  Federated learning (FL) emerges as a popular distributed learning schema that learns from a set of participating users without requiring raw data to be shared. adversarial training (AT) provides a sound solution for centralized learning, extending its usage for FL users has imposed significant challenges. We show that existing FL techniques cannot effectively propagate adversarial robustness among non-iid users. We propose a simple yet effective propagation approach that transfers robustness through carefully designed batch-normalization statistics.
  arXiv  Detail & Related papers  (2021-06-18T15:52:33Z)
• Semi-Federated Learning [4.897245729430157]
  Federated learning (FL) enables massive distributed Information and Communication Technology (ICT) devices to learn a global consensus model without any participants revealing their own data to the central server. We propose the Semi-Federated Learning (Semi-FL) which differs from the FL in two aspects, local clients clustering and in-cluster training. The proposed Semi-FL can be easily applied to future mobile communication networks and require less up-link transmission bandwidth.
  arXiv  Detail & Related papers  (2020-03-28T14:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.