Personalized Federated Learning with Clustered Generalization
- URL: http://arxiv.org/abs/2106.13044v1
- Date: Thu, 24 Jun 2021 14:17:00 GMT
- Title: Personalized Federated Learning with Clustered Generalization
- Authors: Xueyang Tang, Song Guo, Jingcai Guo
- Abstract summary: We study the recent emerging personalized learning (PFL) that aims at dealing with the challenging problem of Non-I.I.D. data in the learning setting.
Key difference between PFL and conventional FL methods in the training target.
We propose a novel concept called clustered generalization to handle the challenge of statistical heterogeneity in FL.
- Score: 16.178571176116073
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We study the recent emerging personalized federated learning (PFL) that aims
at dealing with the challenging problem of Non-I.I.D. data in the federated
learning (FL) setting. The key difference between PFL and conventional FL lies
in the training target, of which the personalized models in PFL usually pursue
a trade-off between personalization (i.e., usually from local models) and
generalization (i.e., usually from the global model) on trained models.
Conventional FL methods can hardly meet this target because of their both
well-developed global and local models. The prevalent PFL approaches usually
maintain a global model to guide the training process of local models and
transfer a proper degree of generalization to them. However, the sole global
model can only provide one direction of generalization and may even transfer
negative effects to some local models when rich statistical diversity exists
across multiple local datasets. Based on our observation, most real or
synthetic data distributions usually tend to be clustered to some degree, of
which we argue different directions of generalization can facilitate the PFL.
In this paper, we propose a novel concept called clustered generalization to
handle the challenge of statistical heterogeneity in FL. Specifically, we
maintain multiple global (generalized) models in the server to associate with
the corresponding amount of local model clusters in clients, and further
formulate the PFL as a bi-level optimization problem that can be solved
efficiently and robustly. We also conduct detailed theoretical analysis and
provide the convergence guarantee for the smooth non-convex objectives.
Experimental results on both synthetic and real datasets show that our approach
surpasses the state-of-the-art by a significant margin.
Related papers
- Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning? [50.03434441234569]
Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing.
While various algorithms have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention.
arXiv Detail & Related papers (2024-09-05T19:00:18Z) - Improving Group Connectivity for Generalization of Federated Deep
Learning [8.594665698279522]
Federated learning (FL) involves multiple clients collaboratively training a global model via iterative local updates and model fusion.
In this paper, we study and improve FL's generalization through a fundamental connectivity'' perspective.
We propose FedGuCci and FedGuCci+, improving group connectivity for better generalization.
arXiv Detail & Related papers (2024-02-29T08:27:01Z) - Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources.
The lack of model privacy protection in FL becomes an unneglectable challenge.
We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
arXiv Detail & Related papers (2023-11-12T11:01:10Z) - PFL-GAN: When Client Heterogeneity Meets Generative Models in
Personalized Federated Learning [55.930403371398114]
We propose a novel generative adversarial network (GAN) sharing and aggregation strategy for personalized learning (PFL)
PFL-GAN addresses the client heterogeneity in different scenarios. More specially, we first learn the similarity among clients and then develop an weighted collaborative data aggregation.
The empirical results through the rigorous experimentation on several well-known datasets demonstrate the effectiveness of PFL-GAN.
arXiv Detail & Related papers (2023-08-23T22:38:35Z) - Tackling Data Heterogeneity in Federated Learning with Class Prototypes [44.746340839025194]
We propose FedNH, a novel method that improves the local models' performance for both personalization and generalization.
We show that imposing uniformity helps to combat prototype collapse while infusing class semantics improves local models.
arXiv Detail & Related papers (2022-12-06T05:15:38Z) - Personalized Federated Learning with Hidden Information on Personalized
Prior [18.8426865970643]
We propose pFedBreD, a framework to solve the problem we model using Bregman divergence regularization.
Our experiments show that our proposal significantly outcompetes other PFL algorithms on multiple public benchmarks.
arXiv Detail & Related papers (2022-11-19T12:45:19Z) - Closing the Gap between Client and Global Model Performance in
Heterogeneous Federated Learning [2.1044900734651626]
We show how the chosen approach for training custom client models has an impact on the global model.
We propose a new approach that combines KD and Learning without Forgetting (LwoF) to produce improved personalised models.
arXiv Detail & Related papers (2022-11-07T11:12:57Z) - FedMR: Fedreated Learning via Model Recombination [7.404225808071622]
Federated Learning (FL) enables global model training across clients without compromising their confidential local data.
Existing FL methods rely on Federated Averaging (FedAvg)-based aggregation.
This paper proposes a novel and effective FL paradigm named FedMR (Federating Model Recombination)
arXiv Detail & Related papers (2022-08-16T11:30:19Z) - Fine-tuning Global Model via Data-Free Knowledge Distillation for
Non-IID Federated Learning [86.59588262014456]
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint.
We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG)
Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
arXiv Detail & Related papers (2022-03-17T11:18:17Z) - Achieving Personalized Federated Learning with Sparse Local Models [75.76854544460981]
Federated learning (FL) is vulnerable to heterogeneously distributed data.
To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user.
Existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory.
We proposeFedSpa, a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.
arXiv Detail & Related papers (2022-01-27T08:43:11Z) - Federated Learning With Quantized Global Model Updates [84.55126371346452]
We study federated learning, which enables mobile devices to utilize their local datasets to train a global model.
We introduce a lossy FL (LFL) algorithm, in which both the global model and the local model updates are quantized before being transmitted.
arXiv Detail & Related papers (2020-06-18T16:55:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.