Bridging the Gap Between Foundation Models and Heterogeneous Federated
Learning
- URL: http://arxiv.org/abs/2310.00247v2
- Date: Wed, 4 Oct 2023 18:27:59 GMT
- Title: Bridging the Gap Between Foundation Models and Heterogeneous Federated
Learning
- Authors: Sixing Yu, J. Pablo Mu\~noz, Ali Jannesari
- Abstract summary: Federated learning (FL) offers privacy-preserving decentralized machine learning, optimizing models at edge clients without sharing private data.
Foundation models (FMs) have gained traction in the artificial intelligence (AI) community due to their exceptional performance across various tasks.
We present an adaptive framework for Resource-aware Federated Foundation Models (RaFFM) to address these challenges.
- Score: 9.198799314774437
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning (FL) offers privacy-preserving decentralized machine
learning, optimizing models at edge clients without sharing private data.
Simultaneously, foundation models (FMs) have gained traction in the artificial
intelligence (AI) community due to their exceptional performance across various
tasks. However, integrating FMs into FL presents challenges, primarily due to
their substantial size and intensive resource requirements. This is especially
true when considering the resource heterogeneity in edge FL systems. We present
an adaptive framework for Resource-aware Federated Foundation Models (RaFFM) to
address these challenges. RaFFM introduces specialized model compression
algorithms tailored for FL scenarios, such as salient parameter prioritization
and high-performance subnetwork extraction. These algorithms enable dynamic
scaling of given transformer-based FMs to fit heterogeneous resource
constraints at the network edge during both FL's optimization and deployment
stages. Experimental results demonstrate that RaFFM shows significant
superiority in resource utilization efficiency and uses fewer resources to
deploy FMs to FL. Despite the lower resource consumption, target models
optimized by RaFFM achieve performance on par with traditional FL methods
applied to full-sized FMs. This is evident across tasks in both natural
language processing and computer vision domains.
Related papers
- FedPFT: Federated Proxy Fine-Tuning of Foundation Models [55.58899993272904]
Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges as a promising strategy for protecting data privacy and valuable FMs.
Existing methods fine-tune FM by allocating sub-FM to clients in FL, leading to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients.
We propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules.
arXiv Detail & Related papers (2024-04-17T16:30:06Z) - FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local
Parameter Sharing [14.938531944702193]
We propose Federated Learning with Local Heterogeneous Sharing (FedLPS)
FedLPS uses transfer learning to facilitate the deployment of multiple tasks on a single device by dividing the local model into a shareable encoder and task-specific encoders.
FedLPS significantly outperforms the state-of-the-art (SOTA) FL frameworks by up to 4.88% and reduces the computational resource consumption by 21.3%.
arXiv Detail & Related papers (2024-02-13T16:30:30Z) - A Survey on Efficient Federated Learning Methods for Foundation Model
Training [66.19763977571114]
Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients.
In the wake of Foundation Models (FM), the reality is different for many deep learning applications.
We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
arXiv Detail & Related papers (2024-01-09T10:22:23Z) - Analysis and Optimization of Wireless Federated Learning with Data
Heterogeneity [72.85248553787538]
This paper focuses on performance analysis and optimization for wireless FL, considering data heterogeneity, combined with wireless resource allocation.
We formulate the loss function minimization problem, under constraints on long-term energy consumption and latency, and jointly optimize client scheduling, resource allocation, and the number of local training epochs (CRE)
Experiments on real-world datasets demonstrate that the proposed algorithm outperforms other benchmarks in terms of the learning accuracy and energy consumption.
arXiv Detail & Related papers (2023-08-04T04:18:01Z) - Vertical Federated Learning over Cloud-RAN: Convergence Analysis and
System Optimization [82.12796238714589]
We propose a novel cloud radio access network (Cloud-RAN) based vertical FL system to enable fast and accurate model aggregation.
We characterize the convergence behavior of the vertical FL algorithm considering both uplink and downlink transmissions.
We establish a system optimization framework by joint transceiver and fronthaul quantization design, for which successive convex approximation and alternate convex search based system optimization algorithms are developed.
arXiv Detail & Related papers (2023-05-04T09:26:03Z) - Joint Age-based Client Selection and Resource Allocation for
Communication-Efficient Federated Learning over NOMA Networks [8.030674576024952]
In federated learning (FL), distributed clients can collaboratively train a shared global model while retaining their own training data locally.
In this paper, a joint optimization problem of client selection and resource allocation is formulated, aiming to minimize the total time consumption of each round in FL over a non-orthogonal multiple access (NOMA) enabled wireless network.
In addition, a server-side artificial neural network (ANN) is proposed to predict the FL models of clients who are not selected at each round to further improve FL performance.
arXiv Detail & Related papers (2023-04-18T13:58:16Z) - Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation
and Convergence [83.58839320635956]
Federated Learning (FL) can be used in mobile edge networks to train machine learning models in a distributed manner.
Recent FL has been interpreted within a Model-Agnostic Meta-Learning (MAML) framework, which brings FL significant advantages in fast adaptation and convergence over heterogeneous datasets.
This paper addresses how much benefit MAML brings to FL and how to maximize such benefit over mobile edge networks.
arXiv Detail & Related papers (2023-03-23T02:42:10Z) - Resource-Aware Heterogeneous Federated Learning using Neural Architecture Search [8.184714897613166]
Federated Learning (FL) is used to train AI/ML models in distributed and privacy-preserving settings.
We propose Resource-aware Federated Learning (RaFL)
RaFL allocates resource-aware specialized models to edge devices using Neural Architecture Search (NAS)
arXiv Detail & Related papers (2022-11-09T09:38:57Z) - Fine-tuning Global Model via Data-Free Knowledge Distillation for
Non-IID Federated Learning [86.59588262014456]
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint.
We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG)
Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
arXiv Detail & Related papers (2022-03-17T11:18:17Z) - Federated Ensemble Model-based Reinforcement Learning in Edge Computing [21.840086997141498]
Federated learning (FL) is a privacy-preserving distributed machine learning paradigm.
We propose a novel FRL algorithm that effectively incorporates model-based RL and ensemble knowledge distillation into FL for the first time.
Specifically, we utilise FL and knowledge distillation to create an ensemble of dynamics models for clients, and then train the policy by solely using the ensemble model without interacting with the environment.
arXiv Detail & Related papers (2021-09-12T16:19:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.