Adaptive Federated Dropout: Improving Communication Efficiency and
Generalization for Federated Learning
- URL: http://arxiv.org/abs/2011.04050v1
- Date: Sun, 8 Nov 2020 18:41:44 GMT
- Title: Adaptive Federated Dropout: Improving Communication Efficiency and
Generalization for Federated Learning
- Authors: Nader Bouacida, Jiahui Hou, Hui Zang and Xin Liu
- Abstract summary: A revolutionary decentralized machine learning setting, known as Federated Learning, enables multiple clients located at different geographical locations to collaboratively learn a machine learning model.
Communication between the clients and the server is considered a main bottleneck in the convergence time of federated learning.
We propose and study Adaptive Federated Dropout (AFD), a novel technique to reduce the communication costs associated with federated learning.
- Score: 6.982736900950362
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: With more regulations tackling users' privacy-sensitive data protection in
recent years, access to such data has become increasingly restricted and
controversial. To exploit the wealth of data generated and located at
distributed entities such as mobile phones, a revolutionary decentralized
machine learning setting, known as Federated Learning, enables multiple clients
located at different geographical locations to collaboratively learn a machine
learning model while keeping all their data on-device. However, the scale and
decentralization of federated learning present new challenges. Communication
between the clients and the server is considered a main bottleneck in the
convergence time of federated learning.
In this paper, we propose and study Adaptive Federated Dropout (AFD), a novel
technique to reduce the communication costs associated with federated learning.
It optimizes both server-client communications and computation costs by
allowing clients to train locally on a selected subset of the global model. We
empirically show that this strategy, combined with existing compression
methods, collectively provides up to 57x reduction in convergence time. It also
outperforms the state-of-the-art solutions for communication efficiency.
Furthermore, it improves model generalization by up to 1.7%.
Related papers
- Communication-Efficient Federated Learning through Adaptive Weight
Clustering and Server-Side Distillation [10.541541376305245]
Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices.
FL is hindered by excessive communication costs due to repeated server-client communication during training.
We propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation.
arXiv Detail & Related papers (2024-01-25T14:49:15Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner.
Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server.
This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
arXiv Detail & Related papers (2023-02-24T08:41:19Z) - Federated Pruning: Improving Neural Network Efficiency with Federated
Learning [24.36174705715827]
We propose Federated Pruning to train a reduced model under the federated setting.
We explore different pruning schemes and provide empirical evidence of the effectiveness of our methods.
arXiv Detail & Related papers (2022-09-14T00:48:37Z) - Federated Learning for Non-IID Data via Client Variance Reduction and
Adaptive Server Update [5.161531917413708]
Federated learning (FL) is an emerging technique used to collaboratively train a global machine learning model.
The main obstacle to FL's practical implementation is the Non-Independent and Identical (Non-IID) data distribution across users.
We propose a method (ComFed) that enhances the whole training process on both the client and server sides.
arXiv Detail & Related papers (2022-07-18T05:58:19Z) - DisPFL: Towards Communication-Efficient Personalized Federated Learning
via Decentralized Sparse Training [84.81043932706375]
We propose a novel personalized federated learning framework in a decentralized (peer-to-peer) communication protocol named Dis-PFL.
Dis-PFL employs personalized sparse masks to customize sparse local models on the edge.
We demonstrate that our method can easily adapt to heterogeneous local clients with varying computation complexities.
arXiv Detail & Related papers (2022-06-01T02:20:57Z) - FedKD: Communication Efficient Federated Learning via Knowledge
Distillation [56.886414139084216]
Federated learning is widely used to learn intelligent models from decentralized data.
In federated learning, clients need to communicate their local model updates in each iteration of model learning.
We propose a communication efficient federated learning method based on knowledge distillation.
arXiv Detail & Related papers (2021-08-30T15:39:54Z) - Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data.
We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks.
We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
arXiv Detail & Related papers (2021-04-27T18:23:54Z) - Federated Residual Learning [53.77128418049985]
We study a new form of federated learning where the clients train personalized local models and make predictions jointly with the server-side shared model.
Using this new federated learning framework, the complexity of the central shared model can be minimized while still gaining all the performance benefits that joint training provides.
arXiv Detail & Related papers (2020-03-28T19:55:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.