One-shot Federated Learning without Server-side Training
- URL: http://arxiv.org/abs/2204.12493v2
- Date: Tue, 9 May 2023 17:40:04 GMT
- Title: One-shot Federated Learning without Server-side Training
- Authors: Shangchao Su, Bin Li, and Xiangyang Xue
- Abstract summary: One-shot federated learning is gaining popularity as a way to reduce communication cost between clients and the server.
Most of the existing one-shot FL methods are based on Knowledge Distillation; however, distillation based approach requires an extra training phase and depends on publicly available data sets or generated pseudo samples.
In this work, we consider a novel and challenging cross-silo setting: performing a single round of parameter aggregation on the local models without server-side training.
- Score: 42.59845771101823
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Federated Learning (FL) has recently made significant progress as a new
machine learning paradigm for privacy protection. Due to the high communication
cost of traditional FL, one-shot federated learning is gaining popularity as a
way to reduce communication cost between clients and the server. Most of the
existing one-shot FL methods are based on Knowledge Distillation; however,
{distillation based approach requires an extra training phase and depends on
publicly available data sets or generated pseudo samples.} In this work, we
consider a novel and challenging cross-silo setting: performing a single round
of parameter aggregation on the local models without server-side training. In
this setting, we propose an effective algorithm for Model Aggregation via
Exploring Common Harmonized Optima (MA-Echo), which iteratively updates the
parameters of all local models to bring them close to a common low-loss area on
the loss surface, without harming performance on their own data sets at the
same time. Compared to the existing methods, MA-Echo can work well even in
extremely non-identical data distribution settings where the support categories
of each local model have no overlapped labels with those of the others. We
conduct extensive experiments on two popular image classification data sets to
compare the proposed method with existing methods and demonstrate the
effectiveness of MA-Echo, which clearly outperforms the state-of-the-arts. The
source code can be accessed in \url{https://github.com/FudanVI/MAEcho}.
Related papers
- TPFL: Tsetlin-Personalized Federated Learning with Confidence-Based Clustering [0.0]
We propose a novel approach called Tsetlin-Personalized Federated Learning.
In this way, models are grouped into clusters based on their confidence towards a specific class.
Clients share only what they are confident about, resulting in the elimination of wrongful weight aggregation.
Results demonstrated that TPFL performance better than baseline methods with 98.94% accuracy on MNIST, 98.52% accuracy on FashionMNIST and 91.16% accuracy on FEMNIST dataset.
arXiv Detail & Related papers (2024-09-16T15:27:35Z) - Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning [51.560590617691005]
We investigate whether it is possible to squeeze more juice" out of each cohort than what is possible in a single communication round.
Our approach leads to up to 74% reduction in the total communication cost needed to train a FL model in the cross-device setting.
arXiv Detail & Related papers (2024-06-03T08:48:49Z) - Exploiting Label Skews in Federated Learning with Model Concatenation [39.38427550571378]
Federated Learning (FL) has emerged as a promising solution to perform deep learning on different data owners without exchanging raw data.
Among different non-IID types, label skews have been challenging and common in image classification and other tasks.
We propose FedConcat, a simple and effective approach that degrades these local models as the base of the global model.
arXiv Detail & Related papers (2023-12-11T10:44:52Z) - Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources.
The lack of model privacy protection in FL becomes an unneglectable challenge.
We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
arXiv Detail & Related papers (2023-11-12T11:01:10Z) - Federated Learning on Non-iid Data via Local and Global Distillation [25.397058380098816]
We propose FedND: federated learning with noise distillation.
In the client, we propose a self-distillation method to train the local model.
In the server, we generate noisy samples for each client and use them to distill other clients.
Experimental results show that the algorithm achieves the best performance and is more communication-efficient than state-of-the-art methods.
arXiv Detail & Related papers (2023-06-26T06:14:01Z) - SalientGrads: Sparse Models for Communication Efficient and Data Aware
Distributed Federated Training [1.0413504599164103]
Federated learning (FL) enables the training of a model leveraging decentralized data in client sites while preserving privacy by not collecting data.
One of the significant challenges of FL is limited computation and low communication bandwidth in resource limited edge client nodes.
We propose Salient Grads, which simplifies the process of sparse training by choosing a data aware subnetwork before training.
arXiv Detail & Related papers (2023-04-15T06:46:37Z) - Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device)
In FL, each data holder trains a model locally and releases it to a central server for aggregation.
In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation).
In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
arXiv Detail & Related papers (2022-11-20T10:49:22Z) - Preserving Privacy in Federated Learning with Ensemble Cross-Domain
Knowledge Distillation [22.151404603413752]
Federated Learning (FL) is a machine learning paradigm where local nodes collaboratively train a central model.
Existing FL methods typically share model parameters or employ co-distillation to address the issue of unbalanced data distribution.
We develop a privacy preserving and communication efficient method in a FL framework with one-shot offline knowledge distillation.
arXiv Detail & Related papers (2022-09-10T05:20:31Z) - An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device.
In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters.
We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
arXiv Detail & Related papers (2021-11-19T12:58:59Z) - A Bayesian Federated Learning Framework with Online Laplace
Approximation [144.7345013348257]
Federated learning allows multiple clients to collaboratively learn a globally shared model.
We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side.
We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
arXiv Detail & Related papers (2021-02-03T08:36:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.