Federated Learning with Label Distribution Skew via Logits Calibration

• URL: http://arxiv.org/abs/2209.00189v1
• Date: Thu, 1 Sep 2022 02:56:39 GMT
• Title: Federated Learning with Label Distribution Skew via Logits Calibration
• Authors: Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Chao Wu
• Abstract summary: In this paper, we investigate the label distribution skew in FL, where the distribution of labels varies across clients. We propose FedLC, which calibrates the logits before softmax cross-entropy according to the probability of occurrence of each class. Experiments on federated datasets and real-world datasets demonstrate that FedLC leads to a more accurate global model.
• Score: 26.98248192651355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional federated optimization methods perform poorly with heterogeneous data (ie, accuracy reduction), especially for highly skewed data. In this paper, we investigate the label distribution skew in FL, where the distribution of labels varies across clients. First, we investigate the label distribution skew from a statistical view. We demonstrate both theoretically and empirically that previous methods based on softmax cross-entropy are not suitable, which can result in local models heavily overfitting to minority classes and missing classes. Additionally, we theoretically introduce a deviation bound to measure the deviation of the gradient after local update. At last, we propose FedLC (\textbf {Fed} erated learning via\textbf {L} ogits\textbf {C} alibration), which calibrates the logits before softmax cross-entropy according to the probability of occurrence of each class. FedLC applies a fine-grained calibrated cross-entropy loss to local update by adding a pairwise label margin. Extensive experiments on federated datasets and real-world datasets demonstrate that FedLC leads to a more accurate global model and much improved performance. Furthermore, integrating other FL methods into our approach can further enhance the performance of the global model.

Related papers

• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels [40.78497779769083]
  Learning with noisy labels can significantly hinder the generalization performance of deep neural networks (DNNs) Existing approaches address this issue through loss correction or example selection methods. We propose the textitDirichlet-based Prediction (DPC) method as a solution.
  arXiv  Detail & Related papers  (2024-01-13T12:33:04Z)
• Calibrated One Round Federated Learning with Bayesian Inference in the Predictive Space [27.259110269667826]
  Federated Learning (FL) involves training a model over a dataset distributed among clients. Small and noisy datasets are common, highlighting the need for well-calibrated models. We propose $beta$-Predictive Bayes, a Bayesian FL algorithm that interpolates between a mixture and product of the predictive posteriors.
  arXiv  Detail & Related papers  (2023-12-15T14:17:16Z)
• Exploiting Label Skews in Federated Learning with Model Concatenation [39.38427550571378]
  Federated Learning (FL) has emerged as a promising solution to perform deep learning on different data owners without exchanging raw data. Among different non-IID types, label skews have been challenging and common in image classification and other tasks. We propose FedConcat, a simple and effective approach that degrades these local models as the base of the global model.
  arXiv  Detail & Related papers  (2023-12-11T10:44:52Z)
• Federated Skewed Label Learning with Logits Fusion [23.062650578266837]
  Federated learning (FL) aims to collaboratively train a shared model across multiple clients without transmitting their local data. We propose FedBalance, which corrects the optimization bias among local models by calibrating their logits. Our method can gain 13% higher average accuracy compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-11-14T14:37:33Z)
• Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
  Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. We find that the difference in logits between the local and global models increases as the model is continuously updated. We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
  arXiv  Detail & Related papers  (2023-08-20T04:41:01Z)
• Graph Out-of-Distribution Generalization with Controllable Data Augmentation [51.17476258673232]
  Graph Neural Network (GNN) has demonstrated extraordinary performance in classifying graph properties. Due to the selection bias of training and testing data, distribution deviation is widespread. We propose OOD calibration to measure the distribution deviation of virtual samples.
  arXiv  Detail & Related papers  (2023-08-16T13:10:27Z)
• Flexible Distribution Alignment: Towards Long-tailed Semi-supervised Learning with Proper Calibration [18.376601653387315]
  Longtailed semi-supervised learning (LTSSL) represents a practical scenario for semi-supervised applications. This problem is often aggravated by discrepancies between labeled and unlabeled class distributions. We introduce Flexible Distribution Alignment (FlexDA), a novel adaptive logit-adjusted loss framework.
  arXiv  Detail & Related papers  (2023-06-07T17:50:59Z)
• Variational Classification [51.2541371924591]
  We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders. Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency. We induce a chosen latent distribution, instead of the implicit assumption found in a standard softmax layer.
  arXiv  Detail & Related papers  (2023-05-17T17:47:19Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Federated Learning from Only Unlabeled Data with Class-Conditional-Sharing Clients [98.22390453672499]
  Supervised federated learning (FL) enables multiple clients to share the trained model without sharing their labeled data. We propose federation of unsupervised learning (FedUL), where the unlabeled data are transformed into surrogate labeled data for each of the clients.
  arXiv  Detail & Related papers  (2022-04-07T09:12:00Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.