One-Shot Federated Learning with Classifier-Guided Diffusion Models
- URL: http://arxiv.org/abs/2311.08870v2
- Date: Thu, 16 Nov 2023 15:43:52 GMT
- Title: One-Shot Federated Learning with Classifier-Guided Diffusion Models
- Authors: Mingzhao Yang, Shangchao Su, Bin Li, Xiangyang Xue
- Abstract summary: One-shot federated learning (OSFL) has gained attention in recent years due to its low communication cost.
In this paper, we explore the novel opportunities that diffusion models bring to OSFL and propose FedCADO.
FedCADO generates data that complies with clients' distributions and subsequently training the aggregated model on the server.
- Score: 44.604485649167216
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: One-shot federated learning (OSFL) has gained attention in recent years due
to its low communication cost. However, most of the existing methods require
auxiliary datasets or training generators, which hinders their practicality in
real-world scenarios. In this paper, we explore the novel opportunities that
diffusion models bring to OSFL and propose FedCADO, utilizing guidance from
client classifiers to generate data that complies with clients' distributions
and subsequently training the aggregated model on the server. Specifically, our
method involves targeted optimizations in two aspects. On one hand, we
conditionally edit the randomly sampled initial noises, embedding them with
specified semantics and distributions, resulting in a significant improvement
in both the quality and stability of generation. On the other hand, we employ
the BN statistics from the classifiers to provide detailed guidance during
generation. These tailored optimizations enable us to limitlessly generate
datasets, which closely resemble the distribution and quality of the original
client dataset. Our method effectively handles the heterogeneous client models
and the problems of non-IID features or labels. In terms of privacy protection,
our method avoids training any generator or transferring any auxiliary
information on clients, eliminating any additional privacy leakage risks.
Leveraging the extensive knowledge stored in the pre-trained diffusion model,
the synthetic datasets can assist us in surpassing the knowledge limitations of
the client samples, resulting in aggregation models that even outperform the
performance ceiling of centralized training in some cases, which is
convincingly demonstrated in the sufficient quantification and visualization
experiments conducted on three large-scale multi-domain image datasets.
Related papers
- Stable Diffusion-based Data Augmentation for Federated Learning with Non-IID Data [9.045647166114916]
Federated Learning (FL) is a promising paradigm for decentralized and collaborative model training.
FL struggles with a significant performance reduction and poor convergence when confronted with Non-Independent and Identically Distributed (Non-IID) data distributions.
We introduce Gen-FedSD, a novel approach that harnesses the powerful capability of state-of-the-art text-to-image foundation models.
arXiv Detail & Related papers (2024-05-13T16:57:48Z) - An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets.
Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round.
We introduce FedAF, a novel aggregation-free FL algorithm.
arXiv Detail & Related papers (2024-04-29T05:55:23Z) - FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm.
It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities.
It achieves substantial and consistent improvements over state-of-the-art baselines.
arXiv Detail & Related papers (2024-02-13T20:04:39Z) - Federated Learning with Projected Trajectory Regularization [65.6266768678291]
Federated learning enables joint training of machine learning models from distributed clients without sharing their local data.
One key challenge in federated learning is to handle non-identically distributed data across the clients.
We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
arXiv Detail & Related papers (2023-12-22T02:12:08Z) - Leveraging Foundation Models to Improve Lightweight Clients in Federated
Learning [16.684749528240587]
Federated Learning (FL) is a distributed training paradigm that enables clients scattered across the world to cooperatively learn a global model without divulging confidential data.
FL faces a significant challenge in the form of heterogeneous data distributions among clients, which leads to a reduction in performance and robustness.
We introduce foundation model distillation to assist in the federated training of lightweight client models and increase their performance under heterogeneous data settings while keeping inference costs low.
arXiv Detail & Related papers (2023-11-14T19:10:56Z) - Adapter-based Selective Knowledge Distillation for Federated
Multi-domain Meeting Summarization [36.916155654985936]
Meeting summarization has emerged as a promising technique for providing users with condensed summaries.
We propose adapter-based Federated Selective Knowledge Distillation (AdaFedSelecKD) for training performant client models.
arXiv Detail & Related papers (2023-08-07T03:34:01Z) - Consistency Regularization for Generalizable Source-free Domain
Adaptation [62.654883736925456]
Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset.
Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets.
We propose a consistency regularization framework to develop a more generalizable SFDA method.
arXiv Detail & Related papers (2023-08-03T07:45:53Z) - Phoenix: A Federated Generative Diffusion Model [6.09170287691728]
Training generative models on large centralized datasets can pose challenges in terms of data privacy, security, and accessibility.
This paper proposes a novel method for training a Denoising Diffusion Probabilistic Model (DDPM) across multiple data sources using Federated Learning (FL) techniques.
arXiv Detail & Related papers (2023-06-07T01:43:09Z) - Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Model [40.83058938096914]
We propose FedDISC, a Federated Diffusion-Inspired Semi-supervised Co-training method.
We first extract prototypes of the labeled server data and use these prototypes to predict pseudo-labels of the client data.
For each category, we compute the cluster centroids and domain-specific representations to signify the semantic and stylistic information of their distributions.
These representations are sent back to the server, which uses the pre-trained to generate synthetic datasets complying with the client distributions and train a global model on it.
arXiv Detail & Related papers (2023-05-06T14:22:33Z) - Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients.
FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification.
Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
arXiv Detail & Related papers (2023-05-01T20:04:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.