Communication-Efficient Adaptive Federated Learning
- URL: http://arxiv.org/abs/2205.02719v3
- Date: Wed, 19 Apr 2023 18:20:06 GMT
- Title: Communication-Efficient Adaptive Federated Learning
- Authors: Yujia Wang, Lu Lin, Jinghui Chen
- Abstract summary: Federated learning is a machine learning paradigm that enables clients to jointly train models without sharing their own localized data.
The implementation of federated learning in practice still faces numerous challenges, such as the large communication overhead.
We propose a novel communication-efficient adaptive learning method (FedCAMS) with theoretical convergence guarantees.
- Score: 17.721884358895686
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning is a machine learning training paradigm that enables
clients to jointly train models without sharing their own localized data.
However, the implementation of federated learning in practice still faces
numerous challenges, such as the large communication overhead due to the
repetitive server-client synchronization and the lack of adaptivity by
SGD-based model updates. Despite that various methods have been proposed for
reducing the communication cost by gradient compression or quantization, and
the federated versions of adaptive optimizers such as FedAdam are proposed to
add more adaptivity, the current federated learning framework still cannot
solve the aforementioned challenges all at once. In this paper, we propose a
novel communication-efficient adaptive federated learning method (FedCAMS) with
theoretical convergence guarantees. We show that in the nonconvex stochastic
optimization setting, our proposed FedCAMS achieves the same convergence rate
of $O(\frac{1}{\sqrt{TKm}})$ as its non-compressed counterparts. Extensive
experiments on various benchmarks verify our theoretical analysis.
Related papers
- Client-Centric Federated Adaptive Optimization [78.30827455292827]
Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private.
We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
arXiv Detail & Related papers (2025-01-17T04:00:50Z) - FADAS: Towards Federated Adaptive Asynchronous Optimization [56.09666452175333]
Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning.
This paper introduces federated adaptive asynchronous optimization, named FADAS, a novel method that incorporates asynchronous updates into adaptive federated optimization with provable guarantees.
We rigorously establish the convergence rate of the proposed algorithms and empirical results demonstrate the superior performance of FADAS over other asynchronous FL baselines.
arXiv Detail & Related papers (2024-07-25T20:02:57Z) - Federated Learning based on Pruning and Recovery [0.0]
This framework integrates asynchronous learning algorithms and pruning techniques.
It addresses the inefficiencies of traditional federated learning algorithms in scenarios involving heterogeneous devices.
It also tackles the staleness issue and inadequate training of certain clients in asynchronous algorithms.
arXiv Detail & Related papers (2024-03-16T14:35:03Z) - Communication-Efficient Federated Learning through Adaptive Weight
Clustering and Server-Side Distillation [10.541541376305245]
Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices.
FL is hindered by excessive communication costs due to repeated server-client communication during training.
We propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation.
arXiv Detail & Related papers (2024-01-25T14:49:15Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner.
Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server.
This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
arXiv Detail & Related papers (2023-02-24T08:41:19Z) - Finite-Time Consensus Learning for Decentralized Optimization with
Nonlinear Gossiping [77.53019031244908]
We present a novel decentralized learning framework based on nonlinear gossiping (NGO), that enjoys an appealing finite-time consensus property to achieve better synchronization.
Our analysis on how communication delay and randomized chats affect learning further enables the derivation of practical variants.
arXiv Detail & Related papers (2021-11-04T15:36:25Z) - CosSGD: Nonlinear Quantization for Communication-efficient Federated
Learning [62.65937719264881]
Federated learning facilitates learning across clients without transferring local data on these clients to a central server.
We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning.
Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
arXiv Detail & Related papers (2020-12-15T12:20:28Z) - Adaptive Federated Optimization [43.78438670284309]
In Federated learning, a large number of clients coordinate with a central server to learn a model without sharing their own data.
adaptive optimization methods have notable success in combating such issues.
We show that the use adaptives can significantly improve the performance of federated learning.
arXiv Detail & Related papers (2020-02-29T16:37:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.