Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availability

• URL: http://arxiv.org/abs/2002.07399v3
• Date: Tue, 22 Dec 2020 02:47:23 GMT
• Title: Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availability
• Authors: Yikai Yan, Chaoyue Niu, Yucheng Ding, Zhenzhe Zheng, Fan Wu, Guihai Chen, Shaojie Tang, Zhihua Wu
• Abstract summary: Federated learning is a new machine learning framework, where a bunch of clients collaboratively train a model without sharing training data. In this work, we consider a practical and issue when deploying federated learning in intermittent mobile environments. We propose a simple distributed nonlinear optimization algorithm, called Federated Latest Averaging (FedLaAvg for short) Our theoretical analysis shows that FedLaAvg attains the convergence rate of $(E1/2/(NT1/2)$, achieving a sublinear speed with respect to the total number of clients.
• Score: 46.85205907718874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning is a new distributed machine learning framework, where a bunch of heterogeneous clients collaboratively train a model without sharing training data. In this work, we consider a practical and ubiquitous issue when deploying federated learning in mobile environments: intermittent client availability, where the set of eligible clients may change during the training process. Such intermittent client availability would seriously deteriorate the performance of the classical Federated Averaging algorithm (FedAvg for short). Thus, we propose a simple distributed non-convex optimization algorithm, called Federated Latest Averaging (FedLaAvg for short), which leverages the latest gradients of all clients, even when the clients are not available, to jointly update the global model in each iteration. Our theoretical analysis shows that FedLaAvg attains the convergence rate of $O(E^{1/2}/(N^{1/4} T^{1/2}))$, achieving a sublinear speedup with respect to the total number of clients. We implement FedLaAvg along with several baselines and evaluate them over the benchmarking MNIST and Sentiment140 datasets. The evaluation results demonstrate that FedLaAvg achieves more stable training than FedAvg in both convex and non-convex settings and indeed reaches a sublinear speedup.

Related papers

• Debiasing Federated Learning with Correlated Client Participation [25.521881752822164]
  This paper introduces a theoretical framework that models client participation in FL as a Markov chain. Every client must wait a minimum number of $R$ rounds (minimum separation) before re-participating. We develop an effective debiasing algorithm for FedAvg that provably converges to the unbiased optimal solution.
  arXiv  Detail & Related papers  (2024-10-02T03:30:53Z)
• Asynchronous Federated Stochastic Optimization for Heterogeneous Objectives Under Arbitrary Delays [0.0]
  Federated learning (FL) was recently proposed to securely train models with data held over multiple locations ("clients") Two major challenges hindering the performance of FL algorithms are long training times caused by straggling clients, and a decline in model accuracy under non-iid local data distributions ("client drift") We propose and analyze Asynchronous Exact Averaging (AREA), a new (sub)gradient algorithm that utilizes communication to speed up convergence and enhance scalability, and employs client memory to correct the client drift caused by variations in client update frequencies.
  arXiv  Detail & Related papers  (2024-05-16T14:22:49Z)
• Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clients [30.135431295658343]
  Federated learning (FL) aims to learn a common global model without exchanging or transferring the data that are stored locally at different clients. In this paper, we propose an efficient federated learning (AFL) framework called DeFedAvg. DeFedAvg is the first AFL algorithm that achieves the desirable linear speedup property, which indicates its high scalability.
  arXiv  Detail & Related papers  (2024-02-17T05:22:46Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms [29.636944156801327]
  Multiple clients collaboratively train one global model without sharing their semantic parsing data. Lorar adjusts each client's contribution to the global model update based on its training loss reduction during each round. Clients with smaller datasets enjoy larger performance gains.
  arXiv  Detail & Related papers  (2023-05-26T19:25:49Z)
• Faster Federated Learning with Decaying Number of Local SGD Steps [23.447883712141422]
  InNIST Learning (FL) devices collaboratively train a machine learning model without sharing their private data with a central or with other clients. In this work we propose $K$ as training progresses, which can jointly improve the final performance of FL model.
  arXiv  Detail & Related papers  (2023-05-16T17:36:34Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• FedAvg with Fine Tuning: Local Updates Lead to Representation Learning [54.65133770989836]
  Federated Averaging (FedAvg) algorithm consists of alternating between a few local gradient updates at client nodes, followed by a model averaging update at the server. We show that the reason behind generalizability of the FedAvg's output is its power in learning the common data representation among the clients' tasks. We also provide empirical evidence demonstrating FedAvg's representation learning ability in federated image classification with heterogeneous data.
  arXiv  Detail & Related papers  (2022-05-27T00:55:24Z)
• A Bayesian Federated Learning Framework with Online Laplace Approximation [144.7345013348257]
  Federated learning allows multiple clients to collaboratively learn a globally shared model. We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
  arXiv  Detail & Related papers  (2021-02-03T08:36:58Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.