A Survey on Efficient Federated Learning Methods for Foundation Model
Training
- URL: http://arxiv.org/abs/2401.04472v2
- Date: Wed, 7 Feb 2024 08:33:18 GMT
- Title: A Survey on Efficient Federated Learning Methods for Foundation Model
Training
- Authors: Herbert Woisetschl\"ager, Alexander Isenko, Shiqiang Wang, Ruben
Mayer, Hans-Arno Jacobsen
- Abstract summary: Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients.
In the wake of Foundation Models (FM), the reality is different for many deep learning applications.
We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
- Score: 66.19763977571114
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Federated Learning (FL) has become an established technique to facilitate
privacy-preserving collaborative training across a multitude of clients.
However, new approaches to FL often discuss their contributions involving small
deep-learning models only and focus on training full models on clients. In the
wake of Foundation Models (FM), the reality is different for many deep learning
applications. Typically, FMs have already been pre-trained across a wide
variety of tasks and can be fine-tuned to specific downstream tasks over
significantly smaller datasets than required for full model training. However,
access to such datasets is often challenging. By its design, FL can help to
open data silos. With this survey, we introduce a novel taxonomy focused on
computational and communication efficiency, the vital elements to make use of
FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient
fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL
frameworks to work with FMs and provide future research opportunities on how to
evaluate generative models in FL as well as the interplay of privacy and PEFT.
Related papers
- Synergizing Foundation Models and Federated Learning: A Survey [23.416321895575507]
This paper discusses the potentials and challenges of synergizing Federated Learning (FL) and Foundation Models (FM)
FL is a collaborative learning paradigm that breaks the barrier of data availability from different participants.
It provides a promising solution to customize and adapt FMs to a wide range of domain-specific tasks using distributed datasets whilst preserving privacy.
arXiv Detail & Related papers (2024-06-18T17:58:09Z) - Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources.
The lack of model privacy protection in FL becomes an unneglectable challenge.
We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
arXiv Detail & Related papers (2023-11-12T11:01:10Z) - Learn From Model Beyond Fine-Tuning: A Survey [78.80920533793595]
Learn From Model (LFM) focuses on the research, modification, and design of foundation models (FM) based on the model interface.
The study of LFM techniques can be broadly categorized into five major areas: model tuning, model distillation, model reuse, meta learning and model editing.
This paper gives a comprehensive review of the current methods based on FM from the perspective of LFM.
arXiv Detail & Related papers (2023-10-12T10:20:36Z) - The Role of Federated Learning in a Wireless World with Foundation Models [59.8129893837421]
Foundation models (FMs) are general-purpose artificial intelligence (AI) models that have recently enabled multiple brand-new generative AI applications.
Currently, the exploration of the interplay between FMs and federated learning (FL) is still in its nascent stage.
This article explores the extent to which FMs are suitable for FL over wireless networks, including a broad overview of research challenges and opportunities.
arXiv Detail & Related papers (2023-10-06T04:13:10Z) - Conquering the Communication Constraints to Enable Large Pre-Trained Models in Federated Learning [13.87836045022214]
Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices.
Recent state-of-the-art pre-trained models are getting more capable but also have more parameters.
Can we find a solution to enable those strong and readily-available pre-trained models in FL to achieve excellent performance while simultaneously reducing the communication burden?
Specifically, we systemically evaluate the performance of FedPEFT across a variety of client stability, data distribution, and differential privacy settings.
arXiv Detail & Related papers (2022-10-04T16:08:54Z) - Federated Learning from Pre-Trained Models: A Contrastive Learning
Approach [43.893267526525904]
Federated Learning (FL) is a machine learning paradigm that allows decentralized clients to learn collaboratively without sharing their private data.
Excessive computation and communication demands pose challenges to current FL frameworks.
We propose a lightweight framework where clients jointly learn to fuse the representations generated by multiple fixed pre-trained models.
arXiv Detail & Related papers (2022-09-21T03:16:57Z) - PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead
of Models -- Federated Learning in Age of Foundation Model [23.916918530195826]
We propose a brand-new FL framework, PromptFL, that replaces the federated model training with the federated prompt training.
PromptFL ships an off-the-shelf FM, i.e., CLIP, to distributed clients who would cooperatively train shared soft prompts.
We empirically analyze the PromptFL via extensive experiments, and show its superiority in terms of system feasibility, user privacy, and performance.
arXiv Detail & Related papers (2022-08-24T15:50:58Z) - On the Importance and Applicability of Pre-Training for Federated
Learning [28.238484580662785]
We conduct a systematic study to explore pre-training for federated learning.
We find that pre-training can improve FL, but also close its accuracy gap to the counterpart centralized learning.
We conclude our paper with an attempt to understand the effect of pre-training on FL.
arXiv Detail & Related papers (2022-06-23T06:02:33Z) - Efficient Split-Mix Federated Learning for On-Demand and In-Situ
Customization [107.72786199113183]
Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data.
In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness.
arXiv Detail & Related papers (2022-03-18T04:58:34Z) - No One Left Behind: Inclusive Federated Learning over Heterogeneous
Devices [79.16481453598266]
We propose InclusiveFL, a client-inclusive federated learning method to handle this problem.
The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities.
We also propose an effective method to share the knowledge among multiple local models with different sizes.
arXiv Detail & Related papers (2022-02-16T13:03:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.