NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clients

• URL: http://arxiv.org/abs/2308.07761v3
• Date: Tue, 10 Sep 2024 16:03:23 GMT
• Title: NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clients
• Authors: Honggu Kang, Seohyeon Cha, Jinwoo Shin, Jongmyeong Lee, Joonhyuk Kang,
• Abstract summary: Federated learning (FL) enables distributed training while preserving data privacy, but stragglers-slow or incapable clients-can significantly slow down the total training time and degrade performance. We propose nested federated learning (NeFL), a framework that efficiently divides deep neural networks into submodels using both depthwise and widthwise scaling. NeFL achieves performance gain, especially for the worst-case submodel compared to baseline approaches.
• Score: 44.89061671579694
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Federated learning (FL) enables distributed training while preserving data privacy, but stragglers-slow or incapable clients-can significantly slow down the total training time and degrade performance. To mitigate the impact of stragglers, system heterogeneity, including heterogeneous computing and network bandwidth, has been addressed. While previous studies have addressed system heterogeneity by splitting models into submodels, they offer limited flexibility in model architecture design, without considering potential inconsistencies arising from training multiple submodel architectures. We propose nested federated learning (NeFL), a generalized framework that efficiently divides deep neural networks into submodels using both depthwise and widthwise scaling. To address the inconsistency arising from training multiple submodel architectures, NeFL decouples a subset of parameters from those being trained for each submodel. An averaging method is proposed to handle these decoupled parameters during aggregation. NeFL enables resource-constrained devices to effectively participate in the FL pipeline, facilitating larger datasets for model training. Experiments demonstrate that NeFL achieves performance gain, especially for the worst-case submodel compared to baseline approaches (7.63% improvement on CIFAR-100). Furthermore, NeFL aligns with recent advances in FL, such as leveraging pre-trained models and accounting for statistical heterogeneity. Our code is available online.

Related papers

• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models. Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• GeFL: Model-Agnostic Federated Learning with Generative Models [3.4546761246181696]
  Federated learning (FL) is a promising paradigm in distributed learning while preserving the privacy of users. We propose Generative Model-Aided Federated Learning (GeFL), incorporating a generative model that aggregates global knowledge across users of heterogeneous models. We empirically demonstrate the consistent performance gains of GeFL-F, while demonstrating better privacy preservation and robustness to a large number of clients.
  arXiv  Detail & Related papers  (2024-12-24T14:39:47Z)
• Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updates [71.81037644563217]
  Synchronous federated learning (FL) is a popular paradigm for collaborative edge learning. As some of the devices may have limited computational resources and varying availability, FL latency is highly sensitive to stragglers. We propose straggler-aware layer-wise federated learning (SALF) that leverages the optimization procedure of NNs via backpropagation to update the global model in a layer-wise fashion.
  arXiv  Detail & Related papers  (2024-03-27T09:14:36Z)
• AdapterFL: Adaptive Heterogeneous Federated Learning for Resource-constrained Mobile Computing Systems [24.013937378054074]
  Federated Learning (FL) enables collaborative learning of large-scale distributed clients without data sharing. Mobile computing systems can only use small low-performance models for collaborative learning. We use a model reassemble strategy to facilitate collaborative training of massive heterogeneous mobile devices adaptively.
  arXiv  Detail & Related papers  (2023-11-23T14:42:43Z)
• Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioning [15.311309249848739]
  Hierarchical independent submodel training (HIST) is a new FL methodology that aims to address these issues in hierarchical cloud-edge-client networks. We demonstrate how HIST can be augmented with over-the-air computation (AirComp) to further enhance the efficiency of the model aggregation over the edge cells.
  arXiv  Detail & Related papers  (2023-10-27T04:42:59Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• pFedSim: Similarity-Aware Model Aggregation Towards Personalized Federated Learning [27.668944118750115]
  federated learning (FL) paradigm emerges to preserve data privacy during model training. One of biggest challenges in FL lies in the non-IID (not identical and independently distributed) data. We propose a novel pFedSim (pFL based on model similarity) algorithm in this work.
  arXiv  Detail & Related papers  (2023-05-25T04:25:55Z)
• Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization [107.72786199113183]
  Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data. In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness.
  arXiv  Detail & Related papers  (2022-03-18T04:58:34Z)
• Ensemble Distillation for Robust Model Fusion in Federated Learning [72.61259487233214]
  Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. We propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients.
  arXiv  Detail & Related papers  (2020-06-12T14:49:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.