FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients

• URL: http://arxiv.org/abs/2201.11865v1
• Date: Fri, 28 Jan 2022 00:09:53 GMT
• Title: FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients
• Authors: Jianyu Wang, Hang Qi, Ankit Singh Rawat, Sashank Reddi, Sagar Waghmare, Felix X. Yu, Gauri Joshi
• Abstract summary: In split learning, only a small part of the model is stored and trained on clients while the remaining large part of the model only stays at the servers. This paper addresses this issue by compressing the additional communication using a novel clustering scheme accompanied by a gradient correction method.
• Score: 41.623518032533035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In classical federated learning, the clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating server. However, updating and communicating the entire model becomes prohibitively expensive when resource-constrained clients collectively aim to train a large machine learning model. Split learning provides a natural solution in such a setting, where only a small part of the model is stored and trained on clients while the remaining large part of the model only stays at the servers. However, the model partitioning employed in split learning introduces a significant amount of communication cost. This paper addresses this issue by compressing the additional communication using a novel clustering scheme accompanied by a gradient correction method. Extensive empirical evaluations on image and text benchmarks show that the proposed method can achieve up to $490\times$ communication cost reduction with minimal drop in accuracy, and enables a desirable performance vs. communication trade-off.

Related papers

• SFPrompt: Communication-Efficient Split Federated Fine-Tuning for Large Pre-Trained Models over Resource-Limited Devices [10.10998320880871]
  SFPrompt is a privacy-preserving fine-tuning method tailored for the federated setting. It combines split learning with federated learning to handle these challenges. Experiments demonstrate that SFPrompt delivers competitive performance as the federated full fine-tuning approach.
  arXiv  Detail & Related papers  (2024-07-24T04:22:37Z)
• Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
  Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources. The lack of model privacy protection in FL becomes an unneglectable challenge. We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
  arXiv  Detail & Related papers  (2023-11-12T11:01:10Z)
• SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Training [1.0413504599164103]
  Federated learning (FL) enables the training of a model leveraging decentralized data in client sites while preserving privacy by not collecting data. One of the significant challenges of FL is limited computation and low communication bandwidth in resource limited edge client nodes. We propose Salient Grads, which simplifies the process of sparse training by choosing a data aware subnetwork before training.
  arXiv  Detail & Related papers  (2023-04-15T06:46:37Z)
• DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training [84.81043932706375]
  We propose a novel personalized federated learning framework in a decentralized (peer-to-peer) communication protocol named Dis-PFL. Dis-PFL employs personalized sparse masks to customize sparse local models on the edge. We demonstrate that our method can easily adapt to heterogeneous local clients with varying computation complexities.
  arXiv  Detail & Related papers  (2022-06-01T02:20:57Z)
• Comfetch: Federated Learning of Large Networks on Constrained Clients via Sketching [28.990067638230254]
  Federated learning (FL) is a popular paradigm for private and collaborative model training on the edge. We propose a novel algorithm, Comdirectional, which allows clients to train large networks using representations of the global neural network.
  arXiv  Detail & Related papers  (2021-09-17T04:48:42Z)
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation [56.886414139084216]
  Federated learning is widely used to learn intelligent models from decentralized data. In federated learning, clients need to communicate their local model updates in each iteration of model learning. We propose a communication efficient federated learning method based on knowledge distillation.
  arXiv  Detail & Related papers  (2021-08-30T15:39:54Z)
• A Bayesian Federated Learning Framework with Online Laplace Approximation [144.7345013348257]
  Federated learning allows multiple clients to collaboratively learn a globally shared model. We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
  arXiv  Detail & Related papers  (2021-02-03T08:36:58Z)
• CosSGD: Nonlinear Quantization for Communication-efficient Federated Learning [62.65937719264881]
  Federated learning facilitates learning across clients without transferring local data on these clients to a central server. We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning. Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
  arXiv  Detail & Related papers  (2020-12-15T12:20:28Z)
• Communication-Efficient Federated Learning via Optimal Client Sampling [20.757477553095637]
  Federated learning (FL) ameliorates privacy concerns in settings where a central server coordinates learning from data distributed across many clients. We propose a novel, simple and efficient way of updating the central model in communication-constrained settings. We test this policy on a synthetic dataset for logistic regression and two FL benchmarks, namely, a classification task on EMNIST and a realistic language modeling task.
  arXiv  Detail & Related papers  (2020-07-30T02:58:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.