One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models

• URL: http://arxiv.org/abs/2311.08870v3
• Date: Fri, 04 Apr 2025 03:46:28 GMT
• Title: One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
• Authors: Mingzhao Yang, Shangchao Su, Bin Li, Xiangyang Xue,
• Abstract summary: FedLMG is a one-shot Federated learning method with Local Model-Guided diffusion models.<n>Clients do not need access to any foundation models but only train and upload their local models.
• Score: 40.83058938096914
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, One-shot Federated Learning methods based on Diffusion Models have garnered increasing attention due to their remarkable performance. However, most of these methods require the deployment of foundation models on client devices, which significantly raises the computational requirements and reduces their adaptability to heterogeneous client models compared to traditional FL methods. In this paper, we propose FedLMG, a heterogeneous one-shot Federated learning method with Local Model-Guided diffusion models. Briefly speaking, in FedLMG, clients do not need access to any foundation models but only train and upload their local models, which is consistent with traditional FL methods. On the clients, we employ classification loss and BN loss to capture the broad category features and detailed contextual features of the client distributions. On the server, based on the uploaded client models, we utilize backpropagation to guide the server's DM in generating synthetic datasets that comply with the client distributions, which are then used to train the aggregated model. By using the locally trained client models as a medium to transfer client knowledge, our method significantly reduces the computational requirements on client devices and effectively adapts to scenarios with heterogeneous clients. Extensive quantitation and visualization experiments on three large-scale real-world datasets, along with theoretical analysis, demonstrate that the synthetic datasets generated by FedLMG exhibit comparable quality and diversity to the client datasets, which leads to an aggregated model that outperforms all compared methods and even the performance ceiling, further elucidating the significant potential of utilizing DMs in FL.

Related papers

• Stable Diffusion-based Data Augmentation for Federated Learning with Non-IID Data [9.045647166114916]
  Federated Learning (FL) is a promising paradigm for decentralized and collaborative model training. FL struggles with a significant performance reduction and poor convergence when confronted with Non-Independent and Identically Distributed (Non-IID) data distributions. We introduce Gen-FedSD, a novel approach that harnesses the powerful capability of state-of-the-art text-to-image foundation models.
  arXiv  Detail & Related papers  (2024-05-13T16:57:48Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
  FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm. It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities. It achieves substantial and consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:04:39Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning [16.684749528240587]
  Federated Learning (FL) is a distributed training paradigm that enables clients scattered across the world to cooperatively learn a global model without divulging confidential data. FL faces a significant challenge in the form of heterogeneous data distributions among clients, which leads to a reduction in performance and robustness. We introduce foundation model distillation to assist in the federated training of lightweight client models and increase their performance under heterogeneous data settings while keeping inference costs low.
  arXiv  Detail & Related papers  (2023-11-14T19:10:56Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization [36.916155654985936]
  Meeting summarization has emerged as a promising technique for providing users with condensed summaries. We propose adapter-based Federated Selective Knowledge Distillation (AdaFedSelecKD) for training performant client models.
  arXiv  Detail & Related papers  (2023-08-07T03:34:01Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Phoenix: A Federated Generative Diffusion Model [6.09170287691728]
  Training generative models on large centralized datasets can pose challenges in terms of data privacy, security, and accessibility. This paper proposes a novel method for training a Denoising Diffusion Probabilistic Model (DDPM) across multiple data sources using Federated Learning (FL) techniques.
  arXiv  Detail & Related papers  (2023-06-07T01:43:09Z)
• Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data [9.792805355704203]
  We study the loss landscape of model averaging in federated learning (FL) We decompose the expected loss of the global model into five factors related to the client models. We propose utilizing IMA on the global model at the late training phase to reduce its deviation from the expected speed.
  arXiv  Detail & Related papers  (2023-05-13T06:19:55Z)
• Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Model [40.83058938096914]
  We propose FedDISC, a Federated Diffusion-Inspired Semi-supervised Co-training method. We first extract prototypes of the labeled server data and use these prototypes to predict pseudo-labels of the client data. For each category, we compute the cluster centroids and domain-specific representations to signify the semantic and stylistic information of their distributions. These representations are sent back to the server, which uses the pre-trained to generate synthetic datasets complying with the client distributions and train a global model on it.
  arXiv  Detail & Related papers  (2023-05-06T14:22:33Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning [2.1044900734651626]
  We show how the chosen approach for training custom client models has an impact on the global model. We propose a new approach that combines KD and Learning without Forgetting (LwoF) to produce improved personalised models.
  arXiv  Detail & Related papers  (2022-11-07T11:12:57Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices [79.16481453598266]
  We propose InclusiveFL, a client-inclusive federated learning method to handle this problem. The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities. We also propose an effective method to share the knowledge among multiple local models with different sizes.
  arXiv  Detail & Related papers  (2022-02-16T13:03:27Z)
• Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learning [8.240098954377794]
  We propose a personalized retrogress-resilient framework to produce a superior personalized model for each client. Our experiments on real-world dermoscopic FL dataset prove that our personalized retrogress-resilient framework outperforms state-of-the-art FL methods.
  arXiv  Detail & Related papers  (2021-10-01T13:24:29Z)
• Federated Multi-Task Learning under a Mixture of Distributions [10.00087964926414]
  Federated Learning (FL) is a framework for on-device collaborative training of machine learning models. First efforts in FL focused on learning a single global model with good average performance across clients, but the global model may be arbitrarily bad for a given client. We study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions.
  arXiv  Detail & Related papers  (2021-08-23T15:47:53Z)
• A Bayesian Federated Learning Framework with Online Laplace Approximation [144.7345013348257]
  Federated learning allows multiple clients to collaboratively learn a globally shared model. We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
  arXiv  Detail & Related papers  (2021-02-03T08:36:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.