The Diversity Bonus: Learning from Dissimilar Distributed Clients in Personalized Federated Learning

• URL: http://arxiv.org/abs/2407.15464v1
• Date: Mon, 22 Jul 2024 08:24:45 GMT
• Title: The Diversity Bonus: Learning from Dissimilar Distributed Clients in Personalized Federated Learning
• Authors: Xinghao Wu, Xuefeng Liu, Jianwei Niu, Guogang Zhu, Shaojie Tang, Xiaotian Li, Jiannong Cao,
• Abstract summary: We propose DiversiFed which allows each client to learn from clients with diversified data distribution. We show that DiversiFed can benefit from dissimilar clients and thus outperform the state-of-the-art methods.
• Score: 20.3260485904085
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized Federated Learning (PFL) is a commonly used framework that allows clients to collaboratively train their personalized models. PFL is particularly useful for handling situations where data from different clients are not independent and identically distributed (non-IID). Previous research in PFL implicitly assumes that clients can gain more benefits from those with similar data distributions. Correspondingly, methods such as personalized weight aggregation are developed to assign higher weights to similar clients during training. We pose a question: can a client benefit from other clients with dissimilar data distributions and if so, how? This question is particularly relevant in scenarios with a high degree of non-IID, where clients have widely different data distributions, and learning from only similar clients will lose knowledge from many other clients. We note that when dealing with clients with similar data distributions, methods such as personalized weight aggregation tend to enforce their models to be close in the parameter space. It is reasonable to conjecture that a client can benefit from dissimilar clients if we allow their models to depart from each other. Based on this idea, we propose DiversiFed which allows each client to learn from clients with diversified data distribution in personalized federated learning. DiversiFed pushes personalized models of clients with dissimilar data distributions apart in the parameter space while pulling together those with similar distributions. In addition, to achieve the above effect without using prior knowledge of data distribution, we design a loss function that leverages the model similarity to determine the degree of attraction and repulsion between any two models. Experiments on several datasets show that DiversiFed can benefit from dissimilar clients and thus outperform the state-of-the-art methods.

Related papers

• Bold but Cautious: Unlocking the Potential of Personalized Federated Learning through Cautiously Aggressive Collaboration [13.857939196296742]
  Key question in personalized federated learning (PFL) is to decide which parameters of a client should be localized or shared with others. This paper introduces a novel guideline for client collaboration in PFL. We propose a new PFL method named FedCAC, which employs a quantitative metric to evaluate each parameter's sensitivity to non-IID data.
  arXiv  Detail & Related papers  (2023-09-20T07:17:28Z)
• PFL-GAN: When Client Heterogeneity Meets Generative Models in Personalized Federated Learning [55.930403371398114]
  We propose a novel generative adversarial network (GAN) sharing and aggregation strategy for personalized learning (PFL) PFL-GAN addresses the client heterogeneity in different scenarios. More specially, we first learn the similarity among clients and then develop an weighted collaborative data aggregation. The empirical results through the rigorous experimentation on several well-known datasets demonstrate the effectiveness of PFL-GAN.
  arXiv  Detail & Related papers  (2023-08-23T22:38:35Z)
• FedSampling: A Better Sampling Strategy for Federated Learning [81.85411484302952]
  Federated learning (FL) is an important technique for learning models from decentralized data in a privacy-preserving way. Existing FL methods usually uniformly sample clients for local model learning in each round. We propose a novel data uniform sampling strategy for federated learning (FedSampling)
  arXiv  Detail & Related papers  (2023-06-25T13:38:51Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Federated Learning of Shareable Bases for Personalization-Friendly Image Classification [54.72892987840267]
  FedBasis learns a set of few shareable basis'' models, which can be linearly combined to form personalized models for clients. Specifically for a new client, only a small set of combination coefficients, not the model weights, needs to be learned. To demonstrate the effectiveness and applicability of FedBasis, we also present a more practical PFL testbed for image classification.
  arXiv  Detail & Related papers  (2023-04-16T20:19:18Z)
• Visual Prompt Based Personalized Federated Learning [83.04104655903846]
  We propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.
  arXiv  Detail & Related papers  (2023-03-15T15:02:15Z)
• Personalized Federated Learning with Multi-branch Architecture [0.0]
  Federated learning (FL) enables multiple clients to collaboratively train models without requiring clients to reveal their raw data to each other. We propose a new PFL method (pFedMB) using multi-branch architecture, which achieves personalization by splitting each layer of a neural network into multiple branches and assigning client-specific weights to each branch. We experimentally show that pFedMB performs better than the state-of-the-art PFL methods using the CIFAR10 and CIFAR100 datasets.
  arXiv  Detail & Related papers  (2022-11-15T06:30:57Z)
• Personalizing or Not: Dynamically Personalized Federated Learning with Incentives [37.42347737911428]
  We propose personalized federated learning (FL) for learning personalized models without sharing private data. We introduce the personalization rate, measured as the fraction of clients willing to train personalized models, into federated settings and propose DyPFL. This technique incentivizes clients to participate in personalizing local models while allowing the adoption of the global model when it performs better.
  arXiv  Detail & Related papers  (2022-08-12T09:51:20Z)
• Unifying Distillation with Personalization in Federated Learning [1.8262547855491458]
  Federated learning (FL) is a decentralized privacy-preserving learning technique in which clients learn a joint collaborative model through a central aggregator without sharing their data. In this setting, all clients learn a single common predictor (FedAvg), which does not generalize well on each client's local data due to the statistical data heterogeneity among clients. In this paper, we address this problem with PersFL, a two-stage personalized learning algorithm. In the first stage, PersFL finds the optimal teacher model of each client during the FL training phase. In the second stage, PersFL distills the useful knowledge from
  arXiv  Detail & Related papers  (2021-05-31T17:54:29Z)
• Personalized Federated Learning with First Order Model Optimization [76.81546598985159]
  We propose an alternative to federated learning, where each client federates with other relevant clients to obtain a stronger model per client-specific objectives. We do not assume knowledge of underlying data distributions or client similarities, and allow each client to optimize for arbitrary target distributions of interest. Our method outperforms existing alternatives, while also enabling new features for personalized FL such as transfer outside of local data distributions.
  arXiv  Detail & Related papers  (2020-12-15T19:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.