Meta Knowledge Condensation for Federated Learning
- URL: http://arxiv.org/abs/2209.14851v1
- Date: Thu, 29 Sep 2022 15:07:37 GMT
- Title: Meta Knowledge Condensation for Federated Learning
- Authors: Ping Liu and Xin Yu and Joey Tianyi Zhou
- Abstract summary: Existing federated learning paradigms usually extensively exchange distributed models at a central solver to achieve a more powerful model.
This would incur severe communication burden between a server and multiple clients especially when data distributions are heterogeneous.
Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the communication cost in federate learning.
- Score: 65.20774786251683
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Existing federated learning paradigms usually extensively exchange
distributed models at a central solver to achieve a more powerful model.
However, this would incur severe communication burden between a server and
multiple clients especially when data distributions are heterogeneous. As a
result, current federated learning methods often require a large number of
communication rounds in training. Unlike existing paradigms, we introduce an
alternative perspective to significantly decrease the communication cost in
federate learning. In this work, we first introduce a meta knowledge
representation method that extracts meta knowledge from distributed clients.
The extracted meta knowledge encodes essential information that can be used to
improve the current model. As the training progresses, the contributions of
training samples to a federated model also vary. Thus, we introduce a dynamic
weight assignment mechanism that enables samples to contribute adaptively to
the current model update. Then, informative meta knowledge from all active
clients is sent to the server for model update. Training a model on the
combined meta knowledge without exposing original data among different clients
can significantly mitigate the heterogeneity issues. Moreover, to further
ameliorate data heterogeneity, we also exchange meta knowledge among clients as
conditional initialization for local meta knowledge extraction. Extensive
experiments demonstrate the effectiveness and efficiency of our proposed
method. Remarkably, our method outperforms the state-of-the-art by a large
margin (from $74.07\%$ to $92.95\%$) on MNIST with a restricted communication
budget (i.e. 10 rounds).
Related papers
- Personalized Federated Learning with Contextual Modulation and
Meta-Learning [2.7716102039510564]
Federated learning has emerged as a promising approach for training machine learning models on decentralized data sources.
We propose a novel framework that combines federated learning with meta-learning techniques to enhance both efficiency and generalization capabilities.
arXiv Detail & Related papers (2023-12-23T08:18:22Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - Adapter-based Selective Knowledge Distillation for Federated
Multi-domain Meeting Summarization [36.916155654985936]
Meeting summarization has emerged as a promising technique for providing users with condensed summaries.
We propose adapter-based Federated Selective Knowledge Distillation (AdaFedSelecKD) for training performant client models.
arXiv Detail & Related papers (2023-08-07T03:34:01Z) - Elastically-Constrained Meta-Learner for Federated Learning [3.032797107899338]
Federated learning is an approach to collaboratively machine learning models for multiple parties that prohibit data sharing.
One of the challenges in federated learning is non-constrained data between clients, as a model can not fit data distribution for all clients.
arXiv Detail & Related papers (2023-06-29T05:58:47Z) - Federated Learning of Neural ODE Models with Different Iteration Counts [0.9444784653236158]
Federated learning is a distributed machine learning approach in which clients train models locally with their own data and upload them to a server so that their trained results are shared between them without uploading raw data to the server.
In this paper, we utilize Neural ODE based models for federated learning.
We show that our approach can reduce communication size by up to 92.4% compared with a baseline ResNet model using CIFAR-10 dataset.
arXiv Detail & Related papers (2022-08-19T17:57:32Z) - FedKD: Communication Efficient Federated Learning via Knowledge
Distillation [56.886414139084216]
Federated learning is widely used to learn intelligent models from decentralized data.
In federated learning, clients need to communicate their local model updates in each iteration of model learning.
We propose a communication efficient federated learning method based on knowledge distillation.
arXiv Detail & Related papers (2021-08-30T15:39:54Z) - Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client.
Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation.
This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
arXiv Detail & Related papers (2021-02-14T05:36:25Z) - CosSGD: Nonlinear Quantization for Communication-efficient Federated
Learning [62.65937719264881]
Federated learning facilitates learning across clients without transferring local data on these clients to a central server.
We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning.
Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
arXiv Detail & Related papers (2020-12-15T12:20:28Z) - WAFFLe: Weight Anonymized Factorization for Federated Learning [88.44939168851721]
In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices.
We propose Weight Anonymized Factorization for Federated Learning (WAFFLe), an approach that combines the Indian Buffet Process with a shared dictionary of weight factors for neural networks.
arXiv Detail & Related papers (2020-08-13T04:26:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.