FedCRL: Personalized Federated Learning with Contrastive Shared Representations for Label Heterogeneity in Non-IID Data

• URL: http://arxiv.org/abs/2404.17916v1
• Date: Sat, 27 Apr 2024 14:05:18 GMT
• Title: FedCRL: Personalized Federated Learning with Contrastive Shared Representations for Label Heterogeneity in Non-IID Data
• Authors: Chenghao Huang, Xiaolu Chen, Yanru Zhang, Hao Wang,
• Abstract summary: This paper proposes a novel personalized Federated Learning algorithm, named Federated Contrastive Learning (FedCRL) FedCRL introduces contrastive representation learning on shared representations to facilitate knowledge acquisition of clients. Our simulations demonstrate FedCRL's effectiveness in mitigating label heterogeneity by achieving accuracy improvements over existing methods.
• Score: 13.146806294562474
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To deal with heterogeneity resulting from label distribution skew and data scarcity in distributed machine learning scenarios, this paper proposes a novel Personalized Federated Learning (PFL) algorithm, named Federated Contrastive Representation Learning (FedCRL). FedCRL introduces contrastive representation learning (CRL) on shared representations to facilitate knowledge acquisition of clients. Specifically, both local model parameters and averaged values of local representations are considered as shareable information to the server, both of which are then aggregated globally. CRL is applied between local representations and global representations to regularize personalized training by drawing similar representations closer and separating dissimilar ones, thereby enhancing local models with external knowledge and avoiding being harmed by label distribution skew. Additionally, FedCRL adopts local aggregation between each local model and the global model to tackle data scarcity. A loss-wise weighting mechanism is introduced to guide the local aggregation using each local model's contrastive loss to coordinate the global model involvement in each client, thus helping clients with scarce data. Our simulations demonstrate FedCRL's effectiveness in mitigating label heterogeneity by achieving accuracy improvements over existing methods on datasets with varying degrees of label heterogeneity.

Related papers

• FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data [46.29190753993415]
  Federated Long-Tailed Learning (Fed-LT) is a paradigm wherein data collected from decentralized local clients manifests a globally prevalent long-tailed distribution. This paper introduces an approach termed Federated Local and Generic Model Training in Fed-LT (FedLoGe), which enhances both local and generic model performance.
  arXiv  Detail & Related papers  (2024-01-17T05:04:33Z)
• FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation [7.052566906745796]
  FedLPA is a layer-wise posterior aggregation method for federated learning. We show that FedLPA significantly improves learning performance over state-of-the-art methods across several metrics.
  arXiv  Detail & Related papers  (2023-09-30T10:51:27Z)
• Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
  Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. We find that the difference in logits between the local and global models increases as the model is continuously updated. We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
  arXiv  Detail & Related papers  (2023-08-20T04:41:01Z)
• Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning [60.41501515192088]
  Federated Learning (FL) has become a popular distributed learning paradigm that involves multiple clients training a global model collaboratively. The data samples usually follow a long-tailed distribution in the real world, and FL on the decentralized and long-tailed data yields a poorly-behaved global model. In this work, we integrate the local real data with the global gradient prototypes to form the local balanced datasets.
  arXiv  Detail & Related papers  (2023-01-25T03:18:10Z)
• Federated Learning with Intermediate Representation Regularization [14.01585596739954]
  Federated learning (FL) enables remote clients to collaboratively train a model without exposing their private data. Previous studies accomplish this by regularizing the distance between the representations learned by the local and global models. We introduce FedIntR, which provides a more fine-grained regularization by integrating the representations of intermediate layers into the local training process.
  arXiv  Detail & Related papers  (2022-10-28T01:43:55Z)
• Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
  Federated learning aims to train models across different clients without the sharing of data for privacy considerations. We study how data heterogeneity affects the representations of the globally aggregated models. We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
  arXiv  Detail & Related papers  (2022-10-01T09:04:17Z)
• Federated and Generalized Person Re-identification through Domain and Feature Hallucinating [88.77196261300699]
  We study the problem of federated domain generalization (FedDG) for person re-identification (re-ID) We propose a novel method, called "Domain and Feature Hallucinating (DFH)", to produce diverse features for learning generalized local and global models. Our method achieves the state-of-the-art performance for FedDG on four large-scale re-ID benchmarks.
  arXiv  Detail & Related papers  (2022-03-05T09:15:13Z)
• Robust Convergence in Federated Learning through Label-wise Clustering [6.693651193181458]
  Non-IID dataset and heterogeneous environment of the local clients are regarded as a major issue in Federated Learning (FL) We propose a novel Label-wise clustering algorithm that guarantees the trainability among geographically heterogeneous local clients. Our paper shows that proposed Label-wise clustering demonstrates prompt and robust convergence compared to other FL algorithms.
  arXiv  Detail & Related papers  (2021-12-28T18:13:09Z)
• Decentralised Person Re-Identification with Selective Knowledge Aggregation [56.40855978874077]
  Existing person re-identification (Re-ID) methods mostly follow a centralised learning paradigm which shares all training data to a collection for model learning. Two recent works have introduced decentralised (federated) Re-ID learning for constructing a globally generalised model (server) However, these methods are poor on how to adapt the generalised model to maximise its performance on individual client domain Re-ID tasks. We present a new Selective Knowledge Aggregation approach to decentralised person Re-ID to optimise the trade-off between model personalisation and generalisation.
  arXiv  Detail & Related papers  (2021-10-21T18:09:53Z)
• GRP-FED: Addressing Client Imbalance in Federated Learning via Global-Regularized Personalization [6.592268037926868]
  We present Global-Regularized Personalization (GRP-FED) to tackle the data imbalanced issue. With adaptive aggregation, the global model treats multiple clients fairly and mitigates the global long-tailed issue. Our results show that our GRP-FED improves under both global and local scenarios.
  arXiv  Detail & Related papers  (2021-08-31T14:09:04Z)
• Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
  We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
  arXiv  Detail & Related papers  (2021-02-14T05:36:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.