FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
- URL: http://arxiv.org/abs/2009.01974v4
- Date: Sun, 10 Oct 2021 18:31:55 GMT
- Title: FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
- Authors: Hong-You Chen, Wei-Lun Chao
- Abstract summary: Federated learning aims to collaboratively train a strong global model by accessing users' locally trained models but not their own data.
A crucial step is therefore to aggregate local models into a global model, which has been shown challenging when users have non-i.i.d. data.
We propose a novel aggregation algorithm named FedBE, which takes a Bayesian inference perspective by sampling higher-quality global models.
- Score: 23.726336635748783
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Federated learning aims to collaboratively train a strong global model by
accessing users' locally trained models but not their own data. A crucial step
is therefore to aggregate local models into a global model, which has been
shown challenging when users have non-i.i.d. data. In this paper, we propose a
novel aggregation algorithm named FedBE, which takes a Bayesian inference
perspective by sampling higher-quality global models and combining them via
Bayesian model Ensemble, leading to much robust aggregation. We show that an
effective model distribution can be constructed by simply fitting a Gaussian or
Dirichlet distribution to the local models. Our empirical studies validate
FedBE's superior performance, especially when users' data are not i.i.d. and
when the neural networks go deeper. Moreover, FedBE is compatible with recent
efforts in regularizing users' model training, making it an easily applicable
module: you only need to replace the aggregation method but leave other parts
of your federated learning algorithm intact. Our code is publicly available at
https://github.com/hongyouc/FedBE.
Related papers
- CoDream: Exchanging dreams instead of models for federated aggregation
with heterogeneous models [8.85591781936764]
We present a novel framework called CoDream, where clients collaboratively optimize randomly data.
Our key insight is that jointly optimizing this data can effectively capture the properties of the global data distribution.
We empirically validate CoDream on standard FL tasks, demonstrating competitive performance despite not sharing model parameters.
arXiv Detail & Related papers (2024-02-25T03:07:32Z) - Federated Learning with Projected Trajectory Regularization [65.6266768678291]
Federated learning enables joint training of machine learning models from distributed clients without sharing their local data.
One key challenge in federated learning is to handle non-identically distributed data across the clients.
We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
arXiv Detail & Related papers (2023-12-22T02:12:08Z) - Exploiting Label Skews in Federated Learning with Model Concatenation [39.38427550571378]
Federated Learning (FL) has emerged as a promising solution to perform deep learning on different data owners without exchanging raw data.
Among different non-IID types, label skews have been challenging and common in image classification and other tasks.
We propose FedConcat, a simple and effective approach that degrades these local models as the base of the global model.
arXiv Detail & Related papers (2023-12-11T10:44:52Z) - Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources.
The lack of model privacy protection in FL becomes an unneglectable challenge.
We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
arXiv Detail & Related papers (2023-11-12T11:01:10Z) - FedSoup: Improving Generalization and Personalization in Federated
Learning via Selective Model Interpolation [32.36334319329364]
Cross-silo federated learning (FL) enables the development of machine learning models on datasets distributed across data centers.
Recent research has found that current FL algorithms face a trade-off between local and global performance when confronted with distribution shifts.
We propose a novel federated model soup method to optimize the trade-off between local and global performance.
arXiv Detail & Related papers (2023-07-20T00:07:29Z) - Fusion of Global and Local Knowledge for Personalized Federated Learning [75.20751492913892]
In this paper, we explore personalized models with low-rank and sparse decomposition.
We propose a two-stage-based algorithm named textbfFederated learning with mixed textbfSparse and textbfRank representation.
Under proper assumptions, we show that the GKR trained by FedSLR can at least sub-linearly converge to a stationary point of the regularized problem.
arXiv Detail & Related papers (2023-02-21T23:09:45Z) - Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.
This creates a barrier to fusing knowledge across individual models to yield a better single model.
We propose a dataless knowledge fusion method that merges models in their parameter space.
arXiv Detail & Related papers (2022-12-19T20:46:43Z) - Personalized Federated Learning with Hidden Information on Personalized
Prior [18.8426865970643]
We propose pFedBreD, a framework to solve the problem we model using Bregman divergence regularization.
Our experiments show that our proposal significantly outcompetes other PFL algorithms on multiple public benchmarks.
arXiv Detail & Related papers (2022-11-19T12:45:19Z) - Learning from aggregated data with a maximum entropy model [73.63512438583375]
We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis.
We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
arXiv Detail & Related papers (2022-10-05T09:17:27Z) - Multi-Center Federated Learning [62.32725938999433]
Federated learning (FL) can protect data privacy in distributed learning.
It merely collects local gradients from users without access to their data.
We propose a novel multi-center aggregation mechanism.
arXiv Detail & Related papers (2021-08-19T12:20:31Z) - Fed-ensemble: Improving Generalization through Model Ensembling in
Federated Learning [5.882234707363695]
Fed-ensemble brings model ensembling to federated learning (FL)
Fed-ensemble can be readily utilized within established FL methods.
arXiv Detail & Related papers (2021-07-21T14:40:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.