Towards Bias Correction of FedAvg over Nonuniform and Time-Varying Communications

• URL: http://arxiv.org/abs/2306.00280v1
• Date: Thu, 1 Jun 2023 01:52:03 GMT
• Title: Towards Bias Correction of FedAvg over Nonuniform and Time-Varying Communications
• Authors: Ming Xiang, Stratis Ioannidis, Edmund Yeh, Carlee Joe-Wong, Lili Su
• Abstract summary: Federated learning (FL) is a decentralized learning framework wherein a parameter server (PS) and a collection of clients collaboratively train a model via a global objective. We show that when the channel conditions are heterogeneous across clients are changing over time, the FedFederated Postponed global model fails to postpone the gossip-type information mixing errors.
• Score: 26.597515045714502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) is a decentralized learning framework wherein a parameter server (PS) and a collection of clients collaboratively train a model via minimizing a global objective. Communication bandwidth is a scarce resource; in each round, the PS aggregates the updates from a subset of clients only. In this paper, we focus on non-convex minimization that is vulnerable to non-uniform and time-varying communication failures between the PS and the clients. Specifically, in each round $t$, the link between the PS and client $i$ is active with probability $p_i^t$, which is $\textit{unknown}$ to both the PS and the clients. This arises when the channel conditions are heterogeneous across clients and are changing over time. We show that when the $p_i^t$'s are not uniform, $\textit{Federated Average}$ (FedAvg) -- the most widely adopted FL algorithm -- fails to minimize the global objective. Observing this, we propose $\textit{Federated Postponed Broadcast}$ (FedPBC) which is a simple variant of FedAvg. It differs from FedAvg in that the PS postpones broadcasting the global model till the end of each round. We show that FedPBC converges to a stationary point of the original objective. The introduced staleness is mild and there is no noticeable slowdown. Both theoretical analysis and numerical results are provided. On the technical front, postponing the global model broadcasts enables implicit gossiping among the clients with active links at round $t$. Despite $p_i^t$'s are time-varying, we are able to bound the perturbation of the global model dynamics via the techniques of controlling the gossip-type information mixing errors.

Related papers

• CAFe: Cost and Age aware Federated Learning [34.16488071014024]
  In many federated learning (FL) models, a common strategy is to wait for at least $M$ clients out of the total $N$ clients to send back their local gradients. We show that the average age of a client at the PS appears explicitly in the theoretical convergence bound, and therefore, can be used as a metric to quantify the convergence of the global model.
  arXiv  Detail & Related papers  (2024-05-24T17:41:30Z)
• Empowering Federated Learning with Implicit Gossiping: Mitigating Connection Unreliability Amidst Unknown and Arbitrary Dynamics [23.466997173249034]
  Federated learning is a popular approach for training a machine learning model without disclosing raw data. We show that when the $p_it$'s vary across clients, the most widely adopted federated learning algorithm, Federated Average (FedAvg), experiences significant bias. We propose Federated Postponed Broadcast (FedPBC), a simple variant of FedAvg.
  arXiv  Detail & Related papers  (2024-04-15T18:58:39Z)
• FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients [50.13097183691517]
  In real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources. We propose a novel federated tuning algorithm, FedRA. In each communication round, FedRA randomly generates an allocation matrix. It reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using adapters.
  arXiv  Detail & Related papers  (2023-11-19T04:43:16Z)
• Adaptive Compression in Federated Learning via Side Information [28.401993810064255]
  We propose a framework that requires approximately $D_KL(q_phi(n) p_theta$ bits of communication. We show that our method can be integrated into many existing compression frameworks to attain the same (and often higher) test accuracy with up to $82$ times smaller than the prior work -- corresponding to 2,650 times overall compression.
  arXiv  Detail & Related papers  (2023-06-22T01:29:50Z)
• Timely Asynchronous Hierarchical Federated Learning: Age of Convergence [59.96266198512243]
  We consider an asynchronous hierarchical federated learning setting with a client-edge-cloud framework. The clients exchange the trained parameters with their corresponding edge servers, which update the locally aggregated model. The goal of each client is to converge to the global model, while maintaining timeliness of the clients.
  arXiv  Detail & Related papers  (2023-06-21T17:39:16Z)
• Federated Learning in the Presence of Adversarial Client Unavailability [16.201377650598516]
  Federated learning is a decentralized machine learning framework that enables collaborative model without revealing raw data. Due to the diverse hardware software limitations, a client may not always be available for the computation requests from the server. In harsh environments like battlefields, adversaries can selectively silence specific clients.
  arXiv  Detail & Related papers  (2023-05-31T15:57:07Z)
• DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics [60.60173139258481]
  Local training on non-iid distributed data results in deflected local optimum. A natural solution is to gather all client data onto the server, such that the server has a global view of the entire data distribution. In this paper, we put forth an idea to collect and leverage global knowledge on the server without hindering data privacy.
  arXiv  Detail & Related papers  (2022-11-20T06:13:06Z)
• $\texttt{FedBC}$: Calibrating Global and Local Models via Federated Learning Beyond Consensus [66.62731854746856]
  In federated learning (FL), the objective of collaboratively learning a global model through aggregation of model updates across devices tends to oppose the goal of personalization via local information. In this work, we calibrate this tradeoff in a quantitative manner through a multi-criterion-based optimization. We demonstrate that $texttFedBC$ balances the global and local model test accuracy metrics across a suite datasets.
  arXiv  Detail & Related papers  (2022-06-22T02:42:04Z)
• Timely Communication in Federated Learning [65.1253801733098]
  We consider a global learning framework in which a parameter server (PS) trains a global model by using $n$ clients without actually storing the client data centrally at a cloud server. Under the proposed scheme, at each iteration, the PS waits for $m$ available clients and sends them the current model. We find the average age of information experienced by each client and numerically characterize the age-optimal $m$ and $k$ values for a given $n$.
  arXiv  Detail & Related papers  (2020-12-31T18:52:08Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.