Exploiting Personalized Invariance for Better Out-of-distribution Generalization in Federated Learning

• URL: http://arxiv.org/abs/2211.11243v1
• Date: Mon, 21 Nov 2022 08:17:03 GMT
• Title: Exploiting Personalized Invariance for Better Out-of-distribution Generalization in Federated Learning
• Authors: Xueyang Tang, Song Guo, Jie Zhang
• Abstract summary: This paper presents a general dual-regularized learning framework to explore the personalized invariance, compared with the exsiting personalized federated learning methods. We show that our method is superior over the existing federated learning and invariant learning methods, in diverse out-of-distribution and Non-IID data cases.
• Score: 13.246981646250518
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, data heterogeneity among the training datasets on the local clients (a.k.a., Non-IID data) has attracted intense interest in Federated Learning (FL), and many personalized federated learning methods have been proposed to handle it. However, the distribution shift between the training dataset and testing dataset on each client is never considered in FL, despite it being general in real-world scenarios. We notice that the distribution shift (a.k.a., out-of-distribution generalization) problem under Non-IID federated setting becomes rather challenging due to the entanglement between personalized and spurious information. To tackle the above problem, we elaborate a general dual-regularized learning framework to explore the personalized invariance, compared with the exsiting personalized federated learning methods which are regularized by a single baseline (usually the global model). Utilizing the personalized invariant features, the developed personalized models can efficiently exploit the most relevant information and meanwhile eliminate spurious information so as to enhance the out-of-distribution generalization performance for each client. Both the theoretical analysis on convergence and OOD generalization performance and the results of extensive experiments demonstrate the superiority of our method over the existing federated learning and invariant learning methods, in diverse out-of-distribution and Non-IID data cases.

Related papers

• Personalized Federated Learning with Adaptive Feature Aggregation and Knowledge Transfer [0.0]
  Federated Learning (FL) is popular as a privacy-preserving machine learning paradigm for generating a single model on decentralized data. We propose a new method personalized Federated learning with Adaptive Feature Aggregation and Knowledge Transfer (FedAFK) We conduct extensive experiments on three datasets in two widely-used heterogeneous settings and show the superior performance of our proposed method over thirteen state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-10-19T11:32:39Z)
• Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalization [35.48757125452761]
  Federated learning is an efficient framework designed to facilitate collaborative model training across multiple distributed devices. A significant challenge of federated learning is data-level heterogeneity, i.e., skewed or long-tailed distribution of private data. We propose a novel Federated Prototype Rectification with Personalization which consists of two parts: Federated Personalization and Federated Prototype Rectification.
  arXiv  Detail & Related papers  (2024-08-15T06:26:46Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Dirichlet-based Uncertainty Quantification for Personalized Federated Learning with Improved Posterior Networks [9.54563359677778]
  This paper presents a new approach to federated learning that allows selecting a model from global and personalized ones. It is achieved through a careful modeling of predictive uncertainties that helps to detect local and global in- and out-of-distribution data. The comprehensive experimental evaluation on the popular real-world image datasets shows the superior performance of the model in the presence of out-of-distribution data.
  arXiv  Detail & Related papers  (2023-12-18T14:30:05Z)
• Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clients [38.804196122833645]
  This paper unveils an information-theoretic generalization framework for Federated Learning. It quantifies generalization errors by evaluating the information entropy of local distributions. Inspired by our deduced generalization bounds, we introduce a weighted aggregation approach and a duo of client selection strategies.
  arXiv  Detail & Related papers  (2023-10-11T03:39:56Z)
• Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning [60.41501515192088]
  Federated Learning (FL) has become a popular distributed learning paradigm that involves multiple clients training a global model collaboratively. The data samples usually follow a long-tailed distribution in the real world, and FL on the decentralized and long-tailed data yields a poorly-behaved global model. In this work, we integrate the local real data with the global gradient prototypes to form the local balanced datasets.
  arXiv  Detail & Related papers  (2023-01-25T03:18:10Z)
• FedGen: Generalizable Federated Learning for Sequential Data [8.784435748969806]
  In many real-world distributed settings, spurious correlations exist due to biases and data sampling issues. We present a generalizable federated learning framework called FedGen, which allows clients to identify and distinguish between spurious and invariant features. We show that FedGen results in models that achieve significantly better generalization and can outperform the accuracy of current federated learning approaches by over 24%.
  arXiv  Detail & Related papers  (2022-11-03T15:48:14Z)
• Straggler-Resilient Personalized Federated Learning [55.54344312542944]
  Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
  arXiv  Detail & Related papers  (2022-06-05T01:14:46Z)
• FEDIC: Federated Learning on Non-IID and Long-Tailed Data via Calibrated Distillation [54.2658887073461]
  Dealing with non-IID data is one of the most challenging problems for federated learning. This paper studies the joint problem of non-IID and long-tailed data in federated learning and proposes a corresponding solution called Federated Ensemble Distillation with Imbalance (FEDIC) FEDIC uses model ensemble to take advantage of the diversity of models trained on non-IID data.
  arXiv  Detail & Related papers  (2022-04-30T06:17:36Z)
• DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixup [58.894901088797376]
  federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data. We propose a general framework to solve the above two challenges simultaneously. We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
  arXiv  Detail & Related papers  (2022-04-16T08:08:29Z)
• Multi-Center Federated Learning [62.57229809407692]
  This paper proposes a novel multi-center aggregation mechanism for federated learning. It learns multiple global models from the non-IID user data and simultaneously derives the optimal matching between users and centers. Our experimental results on benchmark datasets show that our method outperforms several popular federated learning methods.
  arXiv  Detail & Related papers  (2020-05-03T09:14:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.