Spectral Co-Distillation for Personalized Federated Learning
- URL: http://arxiv.org/abs/2401.17124v1
- Date: Mon, 29 Jan 2024 16:01:38 GMT
- Title: Spectral Co-Distillation for Personalized Federated Learning
- Authors: Zihan Chen, Howard H. Yang, Tony Q.S. Quek, Kai Fong Ernest Chong
- Abstract summary: We propose a novel distillation method based on model spectrum information to better capture generic versus personalized representations.
We also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training.
We demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.
- Score: 69.97016362754319
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Personalized federated learning (PFL) has been widely investigated to address
the challenge of data heterogeneity, especially when a single generic model is
inadequate in satisfying the diverse performance requirements of local clients
simultaneously. Existing PFL methods are inherently based on the idea that the
relations between the generic global and personalized local models are captured
by the similarity of model weights. Such a similarity is primarily based on
either partitioning the model architecture into generic versus personalized
components, or modeling client relationships via model weights. To better
capture similar (yet distinct) generic versus personalized model
representations, we propose \textit{spectral distillation}, a novel
distillation method based on model spectrum information. Building upon spectral
distillation, we also introduce a co-distillation framework that establishes a
two-way bridge between generic and personalized model training. Moreover, to
utilize the local idle time in conventional PFL, we propose a wait-free local
training protocol. Through extensive experiments on multiple datasets over
diverse heterogeneous data settings, we demonstrate the outperformance and
efficacy of our proposed spectral co-distillation method, as well as our
wait-free training protocol.
Related papers
- Personalized Federated Learning via Stacking [0.0]
We present a novel personalization approach based on stacked generalization where clients directly send each other privacy-preserving models to be used as base models to train a meta-model on private data.
Our approach is flexible, accommodating various privacy-preserving techniques and model types, and can be applied in horizontal, hybrid, and vertically partitioned federations.
arXiv Detail & Related papers (2024-04-16T23:47:23Z) - Enhancing One-Shot Federated Learning Through Data and Ensemble
Co-Boosting [76.64235084279292]
One-shot Federated Learning (OFL) has become a promising learning paradigm, enabling the training of a global server model via a single communication round.
We introduce a novel framework, Co-Boosting, in which synthesized data and the ensemble model mutually enhance each other progressively.
arXiv Detail & Related papers (2024-02-23T03:15:10Z) - pFedMoE: Data-Level Personalization with Mixture of Experts for
Model-Heterogeneous Personalized Federated Learning [35.72303739409116]
We propose a model-heterogeneous personalized Federated learning with Mixture of Experts (pFedMoE) method.
It assigns a shared homogeneous small feature extractor and a local gating network for each client's local heterogeneous large model.
Overall, pFedMoE enhances local model personalization at a fine-grained data level.
arXiv Detail & Related papers (2024-02-02T12:09:20Z) - PFL-GAN: When Client Heterogeneity Meets Generative Models in
Personalized Federated Learning [55.930403371398114]
We propose a novel generative adversarial network (GAN) sharing and aggregation strategy for personalized learning (PFL)
PFL-GAN addresses the client heterogeneity in different scenarios. More specially, we first learn the similarity among clients and then develop an weighted collaborative data aggregation.
The empirical results through the rigorous experimentation on several well-known datasets demonstrate the effectiveness of PFL-GAN.
arXiv Detail & Related papers (2023-08-23T22:38:35Z) - FedPerfix: Towards Partial Model Personalization of Vision Transformers
in Federated Learning [9.950367271170592]
We investigate where and how to partially personalize a Vision Transformers (ViT) model.
Based on the insights that the self-attention layer and the classification head are the most sensitive parts of a ViT, we propose a novel approach called FedPerfix.
We evaluate the proposed approach on CIFAR-100, OrganAMNIST, and Office-Home datasets and demonstrate its effectiveness compared to several advanced PFL methods.
arXiv Detail & Related papers (2023-08-17T19:22:30Z) - Improving Heterogeneous Model Reuse by Density Estimation [105.97036205113258]
This paper studies multiparty learning, aiming to learn a model using the private data of different participants.
Model reuse is a promising solution for multiparty learning, assuming that a local model has been trained for each party.
arXiv Detail & Related papers (2023-05-23T09:46:54Z) - Visual Prompt Based Personalized Federated Learning [83.04104655903846]
We propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients.
Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.
arXiv Detail & Related papers (2023-03-15T15:02:15Z) - Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.
This creates a barrier to fusing knowledge across individual models to yield a better single model.
We propose a dataless knowledge fusion method that merges models in their parameter space.
arXiv Detail & Related papers (2022-12-19T20:46:43Z) - Personalizing or Not: Dynamically Personalized Federated Learning with
Incentives [37.42347737911428]
We propose personalized federated learning (FL) for learning personalized models without sharing private data.
We introduce the personalization rate, measured as the fraction of clients willing to train personalized models, into federated settings and propose DyPFL.
This technique incentivizes clients to participate in personalizing local models while allowing the adoption of the global model when it performs better.
arXiv Detail & Related papers (2022-08-12T09:51:20Z) - CD$^2$-pFed: Cyclic Distillation-guided Channel Decoupling for Model
Personalization in Federated Learning [24.08509828106899]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to collaboratively learn a shared global model.
We propose CD2-pFed, a novel Cyclic Distillation-guided Channel Decoupling framework, to personalize the global model in FL.
arXiv Detail & Related papers (2022-04-08T07:13:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.