Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decomposition

• URL: http://arxiv.org/abs/2406.19931v2
• Date: Fri, 11 Oct 2024 11:30:01 GMT
• Title: Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decomposition
• Authors: Xinghao Wu, Xuefeng Liu, Jianwei Niu, Haolin Wang, Shaojie Tang, Guogang Zhu, Hao Su,
• Abstract summary: Key strategy of Personalized Federated Learning is to decouple general knowledge (shared among clients) and client-specific knowledge. We introduce FedDecomp, a simple but effective PFL paradigm that employs parameter decomposition additive to address this issue. Experimental results across multiple datasets and varying degrees of data demonstrate that FedDecomp outperforms state-of-the-art methods up to 4.9%.
• Score: 26.218506124446826
• License:
• Abstract: To address data heterogeneity, the key strategy of Personalized Federated Learning (PFL) is to decouple general knowledge (shared among clients) and client-specific knowledge, as the latter can have a negative impact on collaboration if not removed. Existing PFL methods primarily adopt a parameter partitioning approach, where the parameters of a model are designated as one of two types: parameters shared with other clients to extract general knowledge and parameters retained locally to learn client-specific knowledge. However, as these two types of parameters are put together like a jigsaw puzzle into a single model during the training process, each parameter may simultaneously absorb both general and client-specific knowledge, thus struggling to separate the two types of knowledge effectively. In this paper, we introduce FedDecomp, a simple but effective PFL paradigm that employs parameter additive decomposition to address this issue. Instead of assigning each parameter of a model as either a shared or personalized one, FedDecomp decomposes each parameter into the sum of two parameters: a shared one and a personalized one, thus achieving a more thorough decoupling of shared and personalized knowledge compared to the parameter partitioning method. In addition, as we find that retaining local knowledge of specific clients requires much lower model capacity compared with general knowledge across all clients, we let the matrix containing personalized parameters be low rank during the training process. Moreover, a new alternating training strategy is proposed to further improve the performance. Experimental results across multiple datasets and varying degrees of data heterogeneity demonstrate that FedDecomp outperforms state-of-the-art methods up to 4.9\%. The code is available at https://github.com/XinghaoWu/FedDecomp.

Related papers

• pFedGPA: Diffusion-based Generative Parameter Aggregation for Personalized Federated Learning [23.43592558078981]
  Federated Learning (FL) offers a decentralized approach to model training, where data remains local and only model parameters are shared between the clients and the central server. Traditional methods, such as Federated Averaging (FedAvg), linearly aggregate these parameters which are usually trained on heterogeneous data distributions. We propose a novel generative parameter aggregation framework for personalized FL, textttpFedGPA.
  arXiv  Detail & Related papers  (2024-09-09T15:13:56Z)
• Selective Knowledge Sharing for Personalized Federated Learning Under Capacity Heterogeneity [12.333226301343029]
  Pa3dFL is a novel framework designed to enhance local model performance by decoupling and selectively sharing knowledge among capacity-heterogeneous models. We conduct extensive experiments on three datasets to evaluate the effectiveness of Pa3dFL.
  arXiv  Detail & Related papers  (2024-05-31T02:59:25Z)
• Learn What You Need in Personalized Federated Learning [53.83081622573734]
  $textitLearn2pFed$ is a novel algorithm-unrolling-based personalized federated learning framework. We show that $textitLearn2pFed$ significantly outperforms previous personalized federated learning methods.
  arXiv  Detail & Related papers  (2024-01-16T12:45:15Z)
• Bold but Cautious: Unlocking the Potential of Personalized Federated Learning through Cautiously Aggressive Collaboration [13.857939196296742]
  Key question in personalized federated learning (PFL) is to decide which parameters of a client should be localized or shared with others. This paper introduces a novel guideline for client collaboration in PFL. We propose a new PFL method named FedCAC, which employs a quantitative metric to evaluate each parameter's sensitivity to non-IID data.
  arXiv  Detail & Related papers  (2023-09-20T07:17:28Z)
• FedJETs: Efficient Just-In-Time Personalization with Federated Mixture of Experts [48.78037006856208]
  FedJETs is a novel solution by using a Mixture-of-Experts (MoE) framework within a Federated Learning (FL) setup. Our method leverages the diversity of the clients to train specialized experts on different subsets of classes, and a gating function to route the input to the most relevant expert(s) Our approach can improve accuracy up to 18% in state of the art FL settings, while maintaining competitive zero-shot performance.
  arXiv  Detail & Related papers  (2023-06-14T15:47:52Z)
• Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity [8.08257664697228]
  We present a partially personalized formulation of Federated Learning (FL) that strikes a balance between the flexibility of personalization and cooperativeness of global training. In our framework, we split the variables into global parameters, which are shared across all clients, and individual local parameters, which are kept private. We prove that under the right split of parameters, it is possible to find global parameters that allow each client to fit their data perfectly, and refer to the obtained problem as overpersonalized.
  arXiv  Detail & Related papers  (2023-05-29T17:54:50Z)
• Federated Semi-Supervised Learning with Annotation Heterogeneity [57.12560313403097]
  We propose a novel framework called Heterogeneously Annotated Semi-Supervised LEarning (HASSLE) It is a dual-model framework with two models trained separately on labeled and unlabeled data. The dual models can implicitly learn from both types of data across different clients, although each dual model is only trained locally on a single type of data.
  arXiv  Detail & Related papers  (2023-03-04T16:04:49Z)
• Optimizing Server-side Aggregation For Robust Federated Learning via Subspace Training [80.03567604524268]
  Non-IID data distribution across clients and poisoning attacks are two main challenges in real-world federated learning systems. We propose SmartFL, a generic approach that optimize the server-side aggregation process. We provide theoretical analyses of the convergence and generalization capacity for SmartFL.
  arXiv  Detail & Related papers  (2022-11-10T13:20:56Z)
• Straggler-Resilient Personalized Federated Learning [55.54344312542944]
  Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
  arXiv  Detail & Related papers  (2022-06-05T01:14:46Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.