Selective Knowledge Sharing for Personalized Federated Learning Under Capacity Heterogeneity

• URL: http://arxiv.org/abs/2405.20589v1
• Date: Fri, 31 May 2024 02:59:25 GMT
• Title: Selective Knowledge Sharing for Personalized Federated Learning Under Capacity Heterogeneity
• Authors: Zheng Wang, Zheng Wang, Zhaopeng Peng, Zihui Wang, Cheng Wang,
• Abstract summary: Pa3dFL is a novel framework designed to enhance local model performance by decoupling and selectively sharing knowledge among capacity-heterogeneous models. We conduct extensive experiments on three datasets to evaluate the effectiveness of Pa3dFL.
• Score: 12.333226301343029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated Learning (FL) stands to gain significant advantages from collaboratively training capacity-heterogeneous models, enabling the utilization of private data and computing power from low-capacity devices. However, the focus on personalizing capacity-heterogeneous models based on client-specific data has been limited, resulting in suboptimal local model utility, particularly for low-capacity clients. The heterogeneity in both data and device capacity poses two key challenges for model personalization: 1) accurately retaining necessary knowledge embedded within reduced submodels for each client, and 2) effectively sharing knowledge through aggregating size-varying parameters. To this end, we introduce Pa3dFL, a novel framework designed to enhance local model performance by decoupling and selectively sharing knowledge among capacity-heterogeneous models. First, we decompose each layer of the model into general and personal parameters. Then, we maintain uniform sizes for the general parameters across clients and aggregate them through direct averaging. Subsequently, we employ a hyper-network to generate size-varying personal parameters for clients using learnable embeddings. Finally, we facilitate the implicit aggregation of personal parameters by aggregating client embeddings through a self-attention module. We conducted extensive experiments on three datasets to evaluate the effectiveness of Pa3dFL. Our findings indicate that Pa3dFL consistently outperforms baseline methods across various heterogeneity settings. Moreover, Pa3dFL demonstrates competitive communication and computation efficiency compared to baseline approaches, highlighting its practicality and adaptability in adverse system conditions.

Related papers

• Personalized Federated Learning with Adaptive Feature Aggregation and Knowledge Transfer [0.0]
  Federated Learning (FL) is popular as a privacy-preserving machine learning paradigm for generating a single model on decentralized data. We propose a new method personalized Federated learning with Adaptive Feature Aggregation and Knowledge Transfer (FedAFK) We conduct extensive experiments on three datasets in two widely-used heterogeneous settings and show the superior performance of our proposed method over thirteen state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-10-19T11:32:39Z)
• FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization [11.040916982022978]
  Federated Learning (FL) enables collaborative training of machine learning models on decentralized data. Data across clients often differs significantly due to class imbalance, feature distribution skew, sample size imbalance, and other phenomena. We propose a novel Bayesian PFL framework using bi-level optimization to tackle the data heterogeneity challenges.
  arXiv  Detail & Related papers  (2024-05-29T11:28:06Z)
• FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity [82.5448598805968]
  We present an effective and adaptable federated framework FedP3, representing Federated Personalized and Privacy-friendly network Pruning. We offer a theoretical interpretation of FedP3 and its locally differential-private variant, DP-FedP3, and theoretically validate their efficiencies.
  arXiv  Detail & Related papers  (2024-04-15T14:14:05Z)
• One-Shot Federated Learning with Classifier-Guided Diffusion Models [44.604485649167216]
  One-shot federated learning (OSFL) has gained attention in recent years due to its low communication cost. In this paper, we explore the novel opportunities that diffusion models bring to OSFL and propose FedCADO. FedCADO generates data that complies with clients' distributions and subsequently training the aggregated model on the server.
  arXiv  Detail & Related papers  (2023-11-15T11:11:25Z)
• PFL-GAN: When Client Heterogeneity Meets Generative Models in Personalized Federated Learning [55.930403371398114]
  We propose a novel generative adversarial network (GAN) sharing and aggregation strategy for personalized learning (PFL) PFL-GAN addresses the client heterogeneity in different scenarios. More specially, we first learn the similarity among clients and then develop an weighted collaborative data aggregation. The empirical results through the rigorous experimentation on several well-known datasets demonstrate the effectiveness of PFL-GAN.
  arXiv  Detail & Related papers  (2023-08-23T22:38:35Z)
• Efficient Personalized Federated Learning via Sparse Model-Adaptation [47.088124462925684]
  Federated Learning (FL) aims to train machine learning models for multiple clients without sharing their own private data. We propose pFedGate for efficient personalized FL by adaptively and efficiently learning sparse local models. We show that pFedGate achieves superior global accuracy, individual accuracy and efficiency simultaneously over state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T12:21:34Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning [8.420943739336067]
  federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. We introduce a textitquantized and textitpersonalized FL algorithm QuPeD that facilitates collective (personalized model compression) training. Numerically, we validate that QuPeD outperforms competing personalized FL methods, FedAvg, and local training of clients in various heterogeneous settings.
  arXiv  Detail & Related papers  (2021-07-29T10:55:45Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)
• Federated Mutual Learning [65.46254760557073]
  Federated Mutual Leaning (FML) allows clients training a generalized model collaboratively and a personalized model independently. The experiments show that FML can achieve better performance than alternatives in typical Federated learning setting.
  arXiv  Detail & Related papers  (2020-06-27T09:35:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.