What Do We Mean by Generalization in Federated Learning?
- URL: http://arxiv.org/abs/2110.14216v1
- Date: Wed, 27 Oct 2021 07:01:14 GMT
- Title: What Do We Mean by Generalization in Federated Learning?
- Authors: Honglin Yuan, Warren Morningstar, Lin Ning, Karan Singhal
- Abstract summary: Federated learning studies should separate performance gaps from unseen client data.
We propose a framework for disentangling these performance gaps.
We show that dataset synthesis strategy can be important for realistic simulations of generalization.
- Score: 4.3012765978447565
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning data is drawn from a distribution of distributions:
clients are drawn from a meta-distribution, and their data are drawn from local
data distributions. Thus generalization studies in federated learning should
separate performance gaps from unseen client data (out-of-sample gap) from
performance gaps from unseen client distributions (participation gap). In this
work, we propose a framework for disentangling these performance gaps. Using
this framework, we observe and explain differences in behavior across natural
and synthetic federated datasets, indicating that dataset synthesis strategy
can be important for realistic simulations of generalization in federated
learning. We propose a semantic synthesis strategy that enables realistic
simulation without naturally-partitioned data. Informed by our findings, we
call out community suggestions for future federated learning works.
Related papers
- Federated Impression for Learning with Distributed Heterogeneous Data [19.50235109938016]
Federated learning (FL) provides a paradigm that can learn from distributed datasets across clients without requiring them to share data.
In FL, sub-optimal convergence is common among data from different health centers due to the variety in data collection protocols and patient demographics across centers.
We propose FedImpres which alleviates catastrophic forgetting by restoring synthetic data that represents the global information as federated impression.
arXiv Detail & Related papers (2024-09-11T15:37:52Z) - Federated Learning with Projected Trajectory Regularization [65.6266768678291]
Federated learning enables joint training of machine learning models from distributed clients without sharing their local data.
One key challenge in federated learning is to handle non-identically distributed data across the clients.
We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
arXiv Detail & Related papers (2023-12-22T02:12:08Z) - Factor-Assisted Federated Learning for Personalized Optimization with
Heterogeneous Data [6.024145412139383]
Federated learning is an emerging distributed machine learning framework aiming at protecting data privacy.
Data in different clients contain both common knowledge and personalized knowledge.
We develop a novel personalized federated learning framework for heterogeneous data, which we refer to as FedSplit.
arXiv Detail & Related papers (2023-12-07T13:05:47Z) - Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data.
Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
arXiv Detail & Related papers (2023-07-12T16:28:21Z) - Performative Federated Learning: A Solution to Model-Dependent and
Heterogeneous Distribution Shifts [24.196279060605402]
We consider a federated learning (FL) system consisting of multiple clients and a server.
Unlike the conventional FL framework that assumes the client's data is static, we consider scenarios where the clients' data distributions may be reshaped by the deployed decision model.
arXiv Detail & Related papers (2023-05-08T23:29:24Z) - FedIN: Federated Intermediate Layers Learning for Model Heterogeneity [7.781409257429762]
Federated learning (FL) facilitates edge devices to cooperatively train a global shared model while maintaining the training data locally and privately.
In this study, we propose an FL method called Federated Intermediate Layers Learning (FedIN), supporting heterogeneous models without relying on any public dataset.
Experiment results demonstrate the superior performance of FedIN in heterogeneous model environments compared to state-of-the-art algorithms.
arXiv Detail & Related papers (2023-04-03T07:20:43Z) - FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for
Federated Learning on Non-IID Data [69.0785021613868]
Federated learning is a distributed machine learning approach which enables a shared server model to learn by aggregating the locally-computed parameter updates with the training data from spatially-distributed client silos.
We propose the Federated Invariant Learning Consistency (FedILC) approach, which leverages the gradient covariance and the geometric mean of Hessians to capture both inter-silo and intra-silo consistencies.
This is relevant to various fields such as medical healthcare, computer vision, and the Internet of Things (IoT)
arXiv Detail & Related papers (2022-05-19T03:32:03Z) - DRFLM: Distributionally Robust Federated Learning with Inter-client
Noise via Local Mixup [58.894901088797376]
federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data.
We propose a general framework to solve the above two challenges simultaneously.
We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
arXiv Detail & Related papers (2022-04-16T08:08:29Z) - Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client.
Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation.
This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
arXiv Detail & Related papers (2021-02-14T05:36:25Z) - Federated Learning under Importance Sampling [49.17137296715029]
We study the effect of importance sampling and devise schemes for sampling agents and data non-uniformly guided by a performance measure.
We find that in schemes involving sampling without replacement, the performance of the resulting architecture is controlled by two factors related to data variability at each agent.
arXiv Detail & Related papers (2020-12-14T10:08:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.