Federated Learning with Buffered Asynchronous Aggregation
- URL: http://arxiv.org/abs/2106.06639v1
- Date: Fri, 11 Jun 2021 23:29:48 GMT
- Title: Federated Learning with Buffered Asynchronous Aggregation
- Authors: John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Michael
Rabbat, Mani Malek Esmaeili, Dzmitry Huba
- Abstract summary: Federated Learning (FL) trains a shared model across distributed devices while keeping the training data on the devices.
Most FL schemes are synchronous: they perform aggregation of model updates from individual devices.
We propose FedBuff, that combines synchronous and asynchronous FL.
- Score: 0.7327285556439885
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated Learning (FL) trains a shared model across distributed devices
while keeping the training data on the devices. Most FL schemes are
synchronous: they perform a synchronized aggregation of model updates from
individual devices. Synchronous training can be slow because of late-arriving
devices (stragglers). On the other hand, completely asynchronous training makes
FL less private because of incompatibility with secure aggregation. In this
work, we propose a model aggregation scheme, FedBuff, that combines the best
properties of synchronous and asynchronous FL. Similar to synchronous FL,
FedBuff is compatible with secure aggregation. Similar to asynchronous FL,
FedBuff is robust to stragglers. In FedBuff, clients trains asynchronously and
send updates to the server. The server aggregates client updates in a private
buffer until updates have been received, at which point a server model update
is immediately performed. We provide theoretical convergence guarantees for
FedBuff in a non-convex setting. Empirically, FedBuff converges up to 3.8x
faster than previous proposals for synchronous FL (e.g., FedAvgM), and up to
2.5x faster than previous proposals for asynchronous FL (e.g., FedAsync). We
show that FedBuff is robust to different staleness distributions and is more
scalable than synchronous FL techniques.
Related papers
- AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising [49.785626309848276]
AsyncDiff is a universal and plug-and-play acceleration scheme that enables model parallelism across multiple devices.
For the Stable Diffusion v2.1, AsyncDiff achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with only a slight reduction of 0.38 in CLIP Score.
Our experiments also demonstrate that AsyncDiff can be readily applied to video diffusion models with encouraging performances.
arXiv Detail & Related papers (2024-06-11T03:09:37Z) - Enhancing Convergence in Federated Learning: A Contribution-Aware
Asynchronous Approach [0.0]
Federated Learning (FL) is a distributed machine learning paradigm that allows clients to train models on their data while preserving their privacy.
FL algorithms, such as Federated Averaging (FedAvg) and its variants, have been shown to converge well in many scenarios.
However, these methods require clients to upload their local updates to the server in a synchronous manner, which can be slow and unreliable in realistic FL settings.
We propose a contribution-aware asynchronous FL method that takes into account the staleness and statistical heterogeneity of the received updates.
arXiv Detail & Related papers (2024-02-16T12:10:53Z) - Momentum Approximation in Asynchronous Private Federated Learning [26.57367597853813]
momentum approximation can achieve $1.15 textrm--4times$ speed up in convergence compared to existing FLs with momentum.
Momentum approximation can be easily integrated in production FL systems with a minor communication and storage cost.
arXiv Detail & Related papers (2024-02-14T15:35:53Z) - AEDFL: Efficient Asynchronous Decentralized Federated Learning with
Heterogeneous Devices [61.66943750584406]
We propose an Asynchronous Efficient Decentralized FL framework, i.e., AEDFL, in heterogeneous environments.
First, we propose an asynchronous FL system model with an efficient model aggregation method for improving the FL convergence.
Second, we propose a dynamic staleness-aware model update approach to achieve superior accuracy.
Third, we propose an adaptive sparse training method to reduce communication and computation costs without significant accuracy degradation.
arXiv Detail & Related papers (2023-12-18T05:18:17Z) - Federated Nearest Neighbor Machine Translation [66.8765098651988]
In this paper, we propose a novel federated nearest neighbor (FedNN) machine translation framework.
FedNN leverages one-round memorization-based interaction to share knowledge across different clients.
Experiments show that FedNN significantly reduces computational and communication costs compared with FedAvg.
arXiv Detail & Related papers (2023-02-23T18:04:07Z) - FedHiSyn: A Hierarchical Synchronous Federated Learning Framework for
Resource and Data Heterogeneity [56.82825745165945]
Federated Learning (FL) enables training a global model without sharing the decentralized raw data stored on multiple devices to protect data privacy.
We propose a hierarchical synchronous FL framework, i.e., FedHiSyn, to tackle the problems of straggler effects and outdated models.
We evaluate the proposed framework based on MNIST, EMNIST, CIFAR10 and CIFAR100 datasets and diverse heterogeneous settings of devices.
arXiv Detail & Related papers (2022-06-21T17:23:06Z) - A General Theory for Federated Optimization with Asynchronous and
Heterogeneous Clients Updates [10.609815608017065]
We extend the standard FedAvg aggregation scheme by introducing aggregation weights to represent the variability of the clients update time.
Our formalism applies to the general federated setting where clients have heterogeneous datasets and perform at least one step of gradient descent.
We develop in this work FedFix, a novel extension of FedAvg enabling efficient asynchronous federated training while preserving the convergence stability of synchronous aggregation.
arXiv Detail & Related papers (2022-06-21T08:46:05Z) - Papaya: Practical, Private, and Scalable Federated Learning [6.833772874570774]
Cross-device Federated Learning (FL) is a distributed learning paradigm with several challenges.
Most FL systems described in the literature are synchronous - they perform a synchronized aggregation of model updates from individual clients.
In this work, we outline a production asynchronous FL system design.
arXiv Detail & Related papers (2021-11-08T23:46:42Z) - Secure Aggregation for Buffered Asynchronous Federated Learning [4.893896929103367]
Federated learning (FL) typically relies on synchronous training, which is slow due to stragglers.
While asynchronous training handles stragglers efficiently, it does not ensure privacy due to the incompatibility with the secure aggregation protocols.
A buffered asynchronous training protocol known as FedBuff has been proposed recently which bridges the gap between synchronous and asynchronous training to mitigate stragglers and to also ensure privacy simultaneously.
arXiv Detail & Related papers (2021-10-05T17:07:02Z) - Device Scheduling and Update Aggregation Policies for Asynchronous
Federated Learning [72.78668894576515]
Federated Learning (FL) is a newly emerged decentralized machine learning (ML) framework.
We propose an asynchronous FL framework with periodic aggregation to eliminate the straggler issue in FL systems.
arXiv Detail & Related papers (2021-07-23T18:57:08Z) - Over-the-Air Federated Learning from Heterogeneous Data [107.05618009955094]
Federated learning (FL) is a framework for distributed learning of centralized models.
We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local gradient descent (SGD) FL algorithm.
We numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL.
arXiv Detail & Related papers (2020-09-27T08:28:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.