Toward Data Heterogeneity of Federated Learning
- URL: http://arxiv.org/abs/2212.08944v1
- Date: Sat, 17 Dec 2022 20:26:18 GMT
- Title: Toward Data Heterogeneity of Federated Learning
- Abstract summary: We show how data skew and quantity skew will affect the performance of state-of-art federated learning algorithms.
We propose a new algorithm FedMix which adjusts existing federated learning algorithms.
We find that existing state-of-art algorithms such as FedProx and FedNova do not have a significant improvement in all testing cases.
- Score: 4.737991126491218
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning is a popular paradigm for machine learning. Ideally,
federated learning works best when all clients share a similar data
distribution. However, it is not always the case in the real world. Therefore,
the topic of federated learning on heterogeneous data has gained more and more
effort from both academia and industry. In this project, we first do extensive
experiments to show how data skew and quantity skew will affect the performance
of state-of-art federated learning algorithms. Then we propose a new algorithm
FedMix which adjusts existing federated learning algorithms and we show its
performance. We find that existing state-of-art algorithms such as FedProx and
FedNova do not have a significant improvement in all testing cases. But by
testing the existing and new algorithms, it seems that tweaking the client side
is more effective than tweaking the server side.
Related papers
- Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms [54.58593451541316]
We propose the first PU learning benchmark to systematically compare PU learning algorithms.<n>We identify subtle yet critical factors that affect the realistic and fair evaluation of PU learning algorithms.
arXiv Detail & Related papers (2025-09-29T03:13:00Z) - Learning from straggler clients in federated learning [17.536454365543232]
We develop Monte Carlo simulations of client latency that are guided by real-world applications.
We study synchronous optimization algorithms like FedAvg and FedAdam as well as the asynchronous FedBuff algorithm.
Experiments with the EMNIST, CIFAR-100, and StackOverflow benchmark federated learning tasks demonstrate that our new algorithms outperform existing ones in terms of accuracy for straggler clients.
arXiv Detail & Related papers (2024-03-14T04:06:45Z) - A Novel Neural Network-Based Federated Learning System for Imbalanced
and Non-IID Data [2.9642661320713555]
Most machine learning algorithms rely heavily on large amount of data which may be collected from various sources.
To combat this issue, researchers have introduced federated learning, where a prediction model is learnt by ensuring the privacy of data of clients data.
In this research, we propose a centralized, neural network-based federated learning system.
arXiv Detail & Related papers (2023-11-16T17:14:07Z) - FedX: Unsupervised Federated Learning with Cross Knowledge Distillation [70.73383643450548]
FedX is an unsupervised federated learning framework.
It learns unbiased representation from decentralized and heterogeneous local data.
It can be used as an add-on module for existing unsupervised algorithms in federated settings.
arXiv Detail & Related papers (2022-07-19T09:56:44Z) - FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for
Federated Learning on Non-IID Data [69.0785021613868]
Federated learning is a distributed machine learning approach which enables a shared server model to learn by aggregating the locally-computed parameter updates with the training data from spatially-distributed client silos.
We propose the Federated Invariant Learning Consistency (FedILC) approach, which leverages the gradient covariance and the geometric mean of Hessians to capture both inter-silo and intra-silo consistencies.
This is relevant to various fields such as medical healthcare, computer vision, and the Internet of Things (IoT)
arXiv Detail & Related papers (2022-05-19T03:32:03Z) - Improving Fairness via Federated Learning [14.231231094281362]
We propose a new theoretical framework, with which we analyze the value of federated learning in improving fairness.
We then theoretically and empirically show that the performance tradeoff of FedAvg-based fair learning algorithms is strictly worse than that of a fair classifier trained on centralized data.
To resolve this, we propose FedFB, a private fair learning algorithm on decentralized data with a modified FedAvg protocol.
arXiv Detail & Related papers (2021-10-29T05:25:44Z) - FedSEAL: Semi-Supervised Federated Learning with Self-Ensemble Learning
and Negative Learning [7.771967424619346]
Federated learning (FL) is a popular decentralized and privacy-preserving machine learning (FL) framework.
In this paper, we propose a new FL algorithm, called FedSEAL, to solve this Semi-Supervised Federated Learning (SSFL) problem.
Our algorithm utilizes self-ensemble learning and complementary negative learning to enhance both the accuracy and the efficiency of clients' unsupervised learning on unlabeled data.
arXiv Detail & Related papers (2021-10-15T03:03:23Z) - Rethinking Architecture Design for Tackling Data Heterogeneity in
Federated Learning [53.73083199055093]
We show that attention-based architectures (e.g., Transformers) are fairly robust to distribution shifts.
Our experiments show that replacing convolutional networks with Transformers can greatly reduce catastrophic forgetting of previous devices.
arXiv Detail & Related papers (2021-06-10T21:04:18Z) - Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
dataset bias is one of the prevailing causes of unfairness in machine learning.
We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class.
We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
arXiv Detail & Related papers (2021-04-14T14:20:22Z) - Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client.
Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation.
This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
arXiv Detail & Related papers (2021-02-14T05:36:25Z) - Improving Federated Relational Data Modeling via Basis Alignment and
Weight Penalty [18.096788806121754]
Federated learning (FL) has attracted increasing attention in recent years.
We present a modified version of the graph neural network algorithm that performs federated modeling over Knowledge Graph (KG)
We propose a novel optimization algorithm, named FedAlign, with 1) optimal transportation (OT) for on-client personalization and 2) weight constraint to speed up the convergence.
Empirical results show that our proposed method outperforms the state-of-the-art FL methods, such as FedAVG and FedProx, with better convergence.
arXiv Detail & Related papers (2020-11-23T12:52:18Z) - Practical One-Shot Federated Learning for Cross-Silo Setting [114.76232507580067]
One-shot federated learning is a promising approach to make federated learning applicable in cross-silo setting.
We propose a practical one-shot federated learning algorithm named FedKT.
By utilizing the knowledge transfer technique, FedKT can be applied to any classification models and can flexibly achieve differential privacy guarantees.
arXiv Detail & Related papers (2020-10-02T14:09:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.