Communication-Efficient Adaptive Federated Learning
- URL: http://arxiv.org/abs/2205.02719v3
- Date: Wed, 19 Apr 2023 18:20:06 GMT
- Title: Communication-Efficient Adaptive Federated Learning
- Authors: Yujia Wang, Lu Lin, Jinghui Chen
- Abstract summary: Federated learning is a machine learning paradigm that enables clients to jointly train models without sharing their own localized data.
The implementation of federated learning in practice still faces numerous challenges, such as the large communication overhead.
We propose a novel communication-efficient adaptive learning method (FedCAMS) with theoretical convergence guarantees.
- Score: 17.721884358895686
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning is a machine learning training paradigm that enables
clients to jointly train models without sharing their own localized data.
However, the implementation of federated learning in practice still faces
numerous challenges, such as the large communication overhead due to the
repetitive server-client synchronization and the lack of adaptivity by
SGD-based model updates. Despite that various methods have been proposed for
reducing the communication cost by gradient compression or quantization, and
the federated versions of adaptive optimizers such as FedAdam are proposed to
add more adaptivity, the current federated learning framework still cannot
solve the aforementioned challenges all at once. In this paper, we propose a
novel communication-efficient adaptive federated learning method (FedCAMS) with
theoretical convergence guarantees. We show that in the nonconvex stochastic
optimization setting, our proposed FedCAMS achieves the same convergence rate
of $O(\frac{1}{\sqrt{TKm}})$ as its non-compressed counterparts. Extensive
experiments on various benchmarks verify our theoretical analysis.
Related papers
- Federated Learning based on Pruning and Recovery [0.0]
This framework integrates asynchronous learning algorithms and pruning techniques.
It addresses the inefficiencies of traditional federated learning algorithms in scenarios involving heterogeneous devices.
It also tackles the staleness issue and inadequate training of certain clients in asynchronous algorithms.
arXiv Detail & Related papers (2024-03-16T14:35:03Z) - Communication-Efficient Federated Learning through Adaptive Weight
Clustering and Server-Side Distillation [10.541541376305245]
Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices.
FL is hindered by excessive communication costs due to repeated server-client communication during training.
We propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation.
arXiv Detail & Related papers (2024-01-25T14:49:15Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - Combating Exacerbated Heterogeneity for Robust Models in Federated
Learning [91.88122934924435]
Combination of adversarial training and federated learning can lead to the undesired robustness deterioration.
We propose a novel framework called Slack Federated Adversarial Training (SFAT)
We verify the rationality and effectiveness of SFAT on various benchmarked and real-world datasets.
arXiv Detail & Related papers (2023-03-01T06:16:15Z) - Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner.
Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server.
This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
arXiv Detail & Related papers (2023-02-24T08:41:19Z) - Accelerated Federated Learning with Decoupled Adaptive Optimization [53.230515878096426]
federated learning (FL) framework enables clients to collaboratively learn a shared model while keeping privacy of training data on clients.
Recently, many iterations efforts have been made to generalize centralized adaptive optimization methods, such as SGDM, Adam, AdaGrad, etc., to federated settings.
This work aims to develop novel adaptive optimization methods for FL from the perspective of dynamics of ordinary differential equations (ODEs)
arXiv Detail & Related papers (2022-07-14T22:46:43Z) - Finite-Time Consensus Learning for Decentralized Optimization with
Nonlinear Gossiping [77.53019031244908]
We present a novel decentralized learning framework based on nonlinear gossiping (NGO), that enjoys an appealing finite-time consensus property to achieve better synchronization.
Our analysis on how communication delay and randomized chats affect learning further enables the derivation of practical variants.
arXiv Detail & Related papers (2021-11-04T15:36:25Z) - CosSGD: Nonlinear Quantization for Communication-efficient Federated
Learning [62.65937719264881]
Federated learning facilitates learning across clients without transferring local data on these clients to a central server.
We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning.
Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
arXiv Detail & Related papers (2020-12-15T12:20:28Z) - Adaptive Federated Optimization [43.78438670284309]
In Federated learning, a large number of clients coordinate with a central server to learn a model without sharing their own data.
adaptive optimization methods have notable success in combating such issues.
We show that the use adaptives can significantly improve the performance of federated learning.
arXiv Detail & Related papers (2020-02-29T16:37:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.