NeFL: Nested Federated Learning for Heterogeneous Clients
- URL: http://arxiv.org/abs/2308.07761v2
- Date: Mon, 9 Oct 2023 05:46:26 GMT
- Title: NeFL: Nested Federated Learning for Heterogeneous Clients
- Authors: Honggu Kang, Seohyeon Cha, Jinwoo Shin, Jongmyeong Lee, Joonhyuk Kang
- Abstract summary: Federated learning (FL) is a promising approach in distributed learning keeping privacy.
During the training pipeline of FL, slow or incapable clients (i.e., stragglers) slow down the total training time and degrade performance.
We propose nested federated learning (NeFL), a framework that efficiently divides a model into submodels using both depthwise and widthwise scaling.
- Score: 48.160716521203256
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Federated learning (FL) is a promising approach in distributed learning
keeping privacy. However, during the training pipeline of FL, slow or incapable
clients (i.e., stragglers) slow down the total training time and degrade
performance. System heterogeneity, including heterogeneous computing and
network bandwidth, has been addressed to mitigate the impact of stragglers.
Previous studies tackle the system heterogeneity by splitting a model into
submodels, but with less degree-of-freedom in terms of model architecture. We
propose nested federated learning (NeFL), a generalized framework that
efficiently divides a model into submodels using both depthwise and widthwise
scaling. NeFL is implemented by interpreting forward propagation of models as
solving ordinary differential equations (ODEs) with adaptive step sizes. To
address the inconsistency that arises when training multiple submodels of
different architecture, we decouple a few parameters from parameters being
trained for each submodel. NeFL enables resource-constrained clients to
effectively join the FL pipeline and the model to be trained with a larger
amount of data. Through a series of experiments, we demonstrate that NeFL leads
to significant performance gains, especially for the worst-case submodel.
Furthermore, we demonstrate NeFL aligns with recent studies in FL, regarding
pre-trained models of FL and the statistical heterogeneity.
Related papers
- Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updates [71.81037644563217]
Synchronous federated learning (FL) is a popular paradigm for collaborative edge learning.
As some of the devices may have limited computational resources and varying availability, FL latency is highly sensitive to stragglers.
We propose straggler-aware layer-wise federated learning (SALF) that leverages the optimization procedure of NNs via backpropagation to update the global model in a layer-wise fashion.
arXiv Detail & Related papers (2024-03-27T09:14:36Z) - Have Your Cake and Eat It Too: Toward Efficient and Accurate Split Federated Learning [25.47111107054497]
Split Federated Learning (SFL) is promising in AIoT systems.
SFL suffers from the challenges of low inference accuracy and low efficiency.
This paper presents a novel SFL approach, named Sliding Split Federated Learning (S$2$FL)
arXiv Detail & Related papers (2023-11-22T05:09:50Z) - FedOBD: Opportunistic Block Dropout for Efficiently Training Large-scale
Neural Networks through Federated Learning [18.357577491590686]
We propose the Federated Opportunistic Block Dropout (FedOBD) approach to train large-scale neural networks.
FedOBD decomposes large-scale models into semantic blocks so that FL participants can opportunistically upload quantized blocks.
Experiments show that FedOBD reduces the overall communication overhead by more than 88% compared to the best performing baseline approach.
arXiv Detail & Related papers (2022-08-10T06:36:49Z) - Efficient Split-Mix Federated Learning for On-Demand and In-Situ
Customization [107.72786199113183]
Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data.
In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness.
arXiv Detail & Related papers (2022-03-18T04:58:34Z) - Achieving Personalized Federated Learning with Sparse Local Models [75.76854544460981]
Federated learning (FL) is vulnerable to heterogeneously distributed data.
To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user.
Existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory.
We proposeFedSpa, a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.
arXiv Detail & Related papers (2022-01-27T08:43:11Z) - Splitfed learning without client-side synchronization: Analyzing
client-side split network portion size to overall performance [4.689140226545214]
Federated Learning (FL), Split Learning (SL), and SplitFed Learning (SFL) are three recent developments in distributed machine learning.
This paper studies SFL without client-side model synchronization.
It provides only 1%-2% better accuracy than Multi-head Split Learning on the MNIST test set.
arXiv Detail & Related papers (2021-09-19T22:57:23Z) - Ensemble Distillation for Robust Model Fusion in Federated Learning [72.61259487233214]
Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model.
In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side.
We propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients.
arXiv Detail & Related papers (2020-06-12T14:49:47Z) - UVeQFed: Universal Vector Quantization for Federated Learning [179.06583469293386]
Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their possibly private labeled data.
In FL, each user trains its copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model.
We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion.
arXiv Detail & Related papers (2020-06-05T07:10:22Z) - Federated learning with hierarchical clustering of local updates to
improve training on non-IID data [3.3517146652431378]
We show that learning a single joint model is often not optimal in the presence of certain types of non-iid data.
We present a modification to FL by introducing a hierarchical clustering step (FL+HC)
We show how FL+HC allows model training to converge in fewer communication rounds compared to FL without clustering.
arXiv Detail & Related papers (2020-04-24T15:16:01Z) - TiFL: A Tier-based Federated Learning System [17.74678728280232]
Federated Learning (FL) enables learning a shared model across many clients without violating the privacy requirements.
We conduct a case study to show that heterogeneity in resource and data has a significant impact on training time and model accuracy in conventional FL systems.
We propose TiFL, a Tier-based Federated Learning System, which divides clients into tiers based on their training performance and selects clients from the same tier in each training round.
arXiv Detail & Related papers (2020-01-25T01:40:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.