Federated Learning under Covariate Shifts with Generalization Guarantees

• URL: http://arxiv.org/abs/2306.05325v1
• Date: Thu, 8 Jun 2023 16:18:08 GMT
• Title: Federated Learning under Covariate Shifts with Generalization Guarantees
• Authors: Ali Ramezani-Kebrya, Fanghui Liu, Thomas Pethick, Grigorios Chrysos, Volkan Cevher
• Abstract summary: We formulate a new global model training paradigm and propose Federated Importance-Weighted Empirical Risk Minimization (FTW-ERM) We show that FTW-ERM achieves smaller generalization error than classical ERM under certain settings.
• Score: 46.56040078380132
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper addresses intra-client and inter-client covariate shifts in federated learning (FL) with a focus on the overall generalization performance. To handle covariate shifts, we formulate a new global model training paradigm and propose Federated Importance-Weighted Empirical Risk Minimization (FTW-ERM) along with improving density ratio matching methods without requiring perfect knowledge of the supremum over true ratios. We also propose the communication-efficient variant FITW-ERM with the same level of privacy guarantees as those of classical ERM in FL. We theoretically show that FTW-ERM achieves smaller generalization error than classical ERM under certain settings. Experimental results demonstrate the superiority of FTW-ERM over existing FL baselines in challenging imbalanced federated settings in terms of data distribution shifts across clients.

Related papers

• FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averaging [23.786128968778396]
  We find that FedSAM usually performs worse than FedAvg in the case of highly heterogeneous data.<n>We propose a novel and effective federated learning algorithm with Weight Averaging (called texttFedSWA), which aims to find flatter minima.<n>We also introduce a new momentum-based controlled weight averaging FL algorithm (texttFedMoSWA), which is designed to better align local and global models.
  arXiv  Detail & Related papers  (2025-07-26T17:12:40Z)
• Generalization in Federated Learning: A Conditional Mutual Information Framework [45.657352088035516]
  Federated Learning (FL) is a widely adopted privacy-preserving distributed learning framework. We apply an information-theoretic analysis via the conditional mutual information (CMI) framework to study FL's two-level generalization. We derive multiple CMI-based bounds, including hypothesis-based CMI bounds, illustrating how privacy constraints in FL can imply generalization guarantees.
  arXiv  Detail & Related papers  (2025-03-06T04:57:51Z)
• FedImpro: Measuring and Improving Client Update in Federated Learning [77.68805026788836]
  Federated Learning (FL) models often experience client drift caused by heterogeneous data. We present an alternative perspective on client drift and aim to mitigate it by generating improved local models.
  arXiv  Detail & Related papers  (2024-02-10T18:14:57Z)
• Feature Norm Regularized Federated Learning: Transforming Skewed Distributions into Global Insights [16.039822050613022]
  This work introduces the Feature Norm Regularized Federated Learning (FNR-FL) algorithm. FNR-FL incorporates class average feature norms to enhance model accuracy and convergence in non-i.i.d. scenarios. We show that FNR-FL exhibits a substantial 66.24% improvement in accuracy and a significant 11.40% reduction in training time.
  arXiv  Detail & Related papers  (2023-12-12T03:09:37Z)
• Confidence-aware Personalized Federated Learning via Variational Expectation Maximization [34.354154518009956]
  We present a novel framework for personalized Federated Learning (PFL) PFL is a distributed learning scheme to train a shared model across clients. We present a novel framework for PFL based on hierarchical modeling and variational inference.
  arXiv  Detail & Related papers  (2023-05-21T20:12:27Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Adaptive Federated Learning via New Entropy Approach [14.595709494370372]
  Federated Learning (FL) has emerged as a prominent distributed machine learning framework. In this paper, we propose an adaptive FEDerated learning algorithm based on ENTropy theory (FedEnt) to alleviate the parameter deviation among heterogeneous clients.
  arXiv  Detail & Related papers  (2023-03-27T07:57:04Z)
• Beyond ADMM: A Unified Client-variance-reduced Adaptive Federated Learning Framework [82.36466358313025]
  We propose a primal-dual FL algorithm, termed FedVRA, that allows one to adaptively control the variance-reduction level and biasness of the global model. Experiments based on (semi-supervised) image classification tasks demonstrate superiority of FedVRA over the existing schemes.
  arXiv  Detail & Related papers  (2022-12-03T03:27:51Z)
• FL Games: A Federated Learning Framework for Distribution Shifts [71.98708418753786]
  Federated learning aims to train predictive models for data that is distributed across clients, under the orchestration of a server. We propose FL GAMES, a game-theoretic framework for federated learning that learns causal features that are invariant across clients.
  arXiv  Detail & Related papers  (2022-10-31T22:59:03Z)
• FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning [91.74206675452888]
  We propose a novel method FedFM, which guides each client's features to match shared category-wise anchors. To achieve higher efficiency and flexibility, we propose a FedFM variant, called FedFM-Lite, where clients communicate with server with fewer synchronization times and communication bandwidth costs.
  arXiv  Detail & Related papers  (2022-10-14T08:11:34Z)
• Gradient Masked Averaging for Federated Learning [24.687254139644736]
  Federated learning allows a large number of clients with heterogeneous data to coordinate learning of a unified global model. Standard FL algorithms involve averaging of model parameters or gradient updates to approximate the global model at the server. We propose a gradient masked averaging approach for FL as an alternative to the standard averaging of client updates.
  arXiv  Detail & Related papers  (2022-01-28T08:42:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.