FLASH: Federated Learning Across Simultaneous Heterogeneities
- URL: http://arxiv.org/abs/2402.08769v1
- Date: Tue, 13 Feb 2024 20:04:39 GMT
- Title: FLASH: Federated Learning Across Simultaneous Heterogeneities
- Authors: Xiangyu Chang, Sk Miraj Ahmed, Srikanth V. Krishnamurthy, Basak Guler,
Ananthram Swami, Samet Oymak, Amit K. Roy-Chowdhury
- Abstract summary: FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm.
It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities.
It achieves substantial and consistent improvements over state-of-the-art baselines.
- Score: 54.80435317208111
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: The key premise of federated learning (FL) is to train ML models across a
diverse set of data-owners (clients), without exchanging local data. An
overarching challenge to this date is client heterogeneity, which may arise not
only from variations in data distribution, but also in data quality, as well as
compute/communication latency. An integrated view of these diverse and
concurrent sources of heterogeneity is critical; for instance, low-latency
clients may have poor data quality, and vice versa. In this work, we propose
FLASH(Federated Learning Across Simultaneous Heterogeneities), a lightweight
and flexible client selection algorithm that outperforms state-of-the-art FL
frameworks under extensive sources of heterogeneity, by trading-off the
statistical information associated with the client's data quality, data
distribution, and latency. FLASH is the first method, to our knowledge, for
handling all these heterogeneities in a unified manner. To do so, FLASH models
the learning dynamics through contextual multi-armed bandits (CMAB) and
dynamically selects the most promising clients. Through extensive experiments,
we demonstrate that FLASH achieves substantial and consistent improvements over
state-of-the-art baselines -- as much as 10% in absolute accuracy -- thanks to
its unified approach. Importantly, FLASH also outperforms federated aggregation
methods that are designed to handle highly heterogeneous settings and even
enjoys a performance boost when integrated with them.
Related papers
- Non-Federated Multi-Task Split Learning for Heterogeneous Sources [17.47679789733922]
We introduce a new architecture and methodology to perform multi-task learning for heterogeneous data sources efficiently.
We show through theoretical analysis that MTSL can achieve fast convergence by tuning the learning rate of the server and clients.
arXiv Detail & Related papers (2024-05-31T19:27:03Z) - FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization [11.040916982022978]
Federated Learning (FL) enables collaborative training of machine learning models on decentralized data.
Data across clients often differs significantly due to class imbalance, feature distribution skew, sample size imbalance, and other phenomena.
We propose a novel Bayesian PFL framework using bi-level optimization to tackle the data heterogeneity challenges.
arXiv Detail & Related papers (2024-05-29T11:28:06Z) - An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets.
Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round.
We introduce FedAF, a novel aggregation-free FL algorithm.
arXiv Detail & Related papers (2024-04-29T05:55:23Z) - Fake It Till Make It: Federated Learning with Consensus-Oriented
Generation [52.82176415223988]
We propose federated learning with consensus-oriented generation (FedCOG)
FedCOG consists of two key components at the client side: complementary data generation and knowledge-distillation-based model training.
Experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.
arXiv Detail & Related papers (2023-12-10T18:49:59Z) - Leveraging Foundation Models to Improve Lightweight Clients in Federated
Learning [16.684749528240587]
Federated Learning (FL) is a distributed training paradigm that enables clients scattered across the world to cooperatively learn a global model without divulging confidential data.
FL faces a significant challenge in the form of heterogeneous data distributions among clients, which leads to a reduction in performance and robustness.
We introduce foundation model distillation to assist in the federated training of lightweight client models and increase their performance under heterogeneous data settings while keeping inference costs low.
arXiv Detail & Related papers (2023-11-14T19:10:56Z) - FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal
Heterogeneous Federated Learning [37.96957782129352]
We propose a finetuning framework tailored to heterogeneous multi-modal foundation models, called Federated Dual-Aadapter Teacher (Fed DAT)
Fed DAT addresses data heterogeneity by regularizing the client local updates and applying Mutual Knowledge Distillation (MKD) for an efficient knowledge transfer.
To demonstrate its effectiveness, we conduct extensive experiments on four multi-modality FL benchmarks with different types of data heterogeneity.
arXiv Detail & Related papers (2023-08-21T21:57:01Z) - Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training.
In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework.
Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
arXiv Detail & Related papers (2023-08-11T09:58:47Z) - FL Games: A Federated Learning Framework for Distribution Shifts [71.98708418753786]
Federated learning aims to train predictive models for data that is distributed across clients, under the orchestration of a server.
We propose FL GAMES, a game-theoretic framework for federated learning that learns causal features that are invariant across clients.
arXiv Detail & Related papers (2022-10-31T22:59:03Z) - Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data.
We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks.
We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
arXiv Detail & Related papers (2021-04-27T18:23:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.