Communication-Efficient Federated Learning With Data and Client Heterogeneity

• URL: http://arxiv.org/abs/2206.10032v3
• Date: Sat, 3 Jun 2023 15:22:55 GMT
• Title: Communication-Efficient Federated Learning With Data and Client Heterogeneity
• Authors: Hossein Zakerinia, Shayan Talaei, Giorgi Nadiradze, Dan Alistarh
• Abstract summary: Federated Learning (FL) enables large-scale distributed training of machine learning models. executing FL at scale comes with inherent practical challenges. We present the first variant of the classic federated averaging (FedAvg) algorithm.
• Score: 22.432529149142976
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) enables large-scale distributed training of machine learning models, while still allowing individual nodes to maintain data locally. However, executing FL at scale comes with inherent practical challenges: 1) heterogeneity of the local node data distributions, 2) heterogeneity of node computational speeds (asynchrony), but also 3) constraints in the amount of communication between the clients and the server. In this work, we present the first variant of the classic federated averaging (FedAvg) algorithm which, at the same time, supports data heterogeneity, partial client asynchrony, and communication compression. Our algorithm comes with a rigorous analysis showing that, in spite of these system relaxations, it can provide similar convergence to FedAvg in interesting parameter regimes. Experimental results in the rigorous LEAF benchmark on setups of up to $300$ nodes show that our algorithm ensures fast convergence for standard federated tasks, improving upon prior quantized and asynchronous approaches.

Related papers

• Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration [66.43954501171292]
  We introduce Catalyst Acceleration and propose an acceleration Decentralized Federated Learning algorithm called DFedCata. DFedCata consists of two main components: the Moreau envelope function, which addresses parameter inconsistencies, and Nesterov's extrapolation step, which accelerates the aggregation phase. Empirically, we demonstrate the advantages of the proposed algorithm in both convergence speed and generalization performance on CIFAR10/100 with various non-iid data distributions.
  arXiv  Detail & Related papers  (2024-10-09T06:17:16Z)
• Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clients [30.135431295658343]
  Federated learning (FL) aims to learn a common global model without exchanging or transferring the data that are stored locally at different clients. In this paper, we propose an efficient federated learning (AFL) framework called DeFedAvg. DeFedAvg is the first AFL algorithm that achieves the desirable linear speedup property, which indicates its high scalability.
  arXiv  Detail & Related papers  (2024-02-17T05:22:46Z)
• FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices using a Computing Power Aware Scheduler [5.550660753625296]
  Cross-silo federated learning offers a promising solution to collaboratively train AI models without compromising privacy of local datasets. In this paper, we introduce an innovative semi-aware Fedasynchronous federated learning algorithm with a computing power scheduler on the server side. We demonstrate that Fed achieves faster convergence and accuracy than other algorithms when performing federated learning on higher clients.
  arXiv  Detail & Related papers  (2023-09-26T05:03:13Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Momentum Benefits Non-IID Federated Learning Simply and Provably [22.800862422479913]
  Federated learning is a powerful paradigm for large-scale machine learning. FedAvg and SCAFFOLD are two prominent algorithms to address these challenges. This paper explores the utilization of momentum to enhance the performance of FedAvg and SCAFFOLD.
  arXiv  Detail & Related papers  (2023-06-28T18:52:27Z)
• Straggler-Resilient Personalized Federated Learning [55.54344312542944]
  Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
  arXiv  Detail & Related papers  (2022-06-05T01:14:46Z)
• Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients [19.496278017418113]
  Federated Learning (FL) allows training machine learning models in privacy-constrained scenarios by enabling the cooperation of edge devices without requiring local data sharing. This approach raises several challenges due to the different statistical distribution of the local datasets and the clients' computational heterogeneity. We propose FedSeq, a novel framework leveraging the sequential training of subgroups of heterogeneous clients, i.e. superclients, to emulate the centralized paradigm in a privacy-compliant way.
  arXiv  Detail & Related papers  (2022-01-26T12:33:23Z)
• Two-Bit Aggregation for Communication Efficient and Differentially Private Federated Learning [79.66767935077925]
  In federated learning (FL), a machine learning model is trained on multiple nodes in a decentralized manner, while keeping the data local and not shared with other nodes. The information sent from the nodes to the server may reveal some details about each node's local data, thus raising privacy concerns. A novel two-bit aggregation algorithm is proposed with guaranteed differential privacy and reduced uplink communication overhead.
  arXiv  Detail & Related papers  (2021-10-06T19:03:58Z)
• Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity [57.275753974812666]
  Federated learning involves learning from data samples distributed across a network of clients while the data remains local. In this paper, we propose a novel straggler-resilient federated learning method that incorporates statistical characteristics of the clients' data to adaptively select the clients in order to speed up the learning procedure.
  arXiv  Detail & Related papers  (2020-12-28T19:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.