Accelerating Non-IID Federated Learning via Heterogeneity-Guided Client
Sampling
- URL: http://arxiv.org/abs/2310.00198v1
- Date: Sat, 30 Sep 2023 00:29:30 GMT
- Title: Accelerating Non-IID Federated Learning via Heterogeneity-Guided Client
Sampling
- Authors: Huancheng Chen, Haris Vikalo
- Abstract summary: HiCS-FL is a novel client selection method in which the server estimates statistical heterogeneity of a client's data using the client's update of the network's output layer.
In non-IID settings HiCS-FL achieves faster convergence and lower training variance than state-of-the-art FL client selection schemes.
- Score: 17.56259695496955
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Statistical heterogeneity of data present at client devices in a federated
learning (FL) system renders the training of a global model in such systems
difficult. Particularly challenging are the settings where due to resource
constraints only a small fraction of clients can participate in any given round
of FL. Recent approaches to training a global model in FL systems with non-IID
data have focused on developing client selection methods that aim to sample
clients with more informative updates of the model. However, existing client
selection techniques either introduce significant computation overhead or
perform well only in the scenarios where clients have data with similar
heterogeneity profiles. In this paper, we propose HiCS-FL (Federated Learning
via Hierarchical Clustered Sampling), a novel client selection method in which
the server estimates statistical heterogeneity of a client's data using the
client's update of the network's output layer and relies on this information to
cluster and sample the clients. We analyze the ability of the proposed
techniques to compare heterogeneity of different datasets, and characterize
convergence of the training process that deploys the introduced client
selection method. Extensive experimental results demonstrate that in non-IID
settings HiCS-FL achieves faster convergence and lower training variance than
state-of-the-art FL client selection schemes. Notably, HiCS-FL drastically
reduces computation cost compared to existing selection schemes and is
adaptable to different heterogeneity scenarios.
Related papers
- FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering [26.478852701376294]
Federated learning (FL) is an emerging distributed machine learning paradigm.
One of the major challenges in FL is the presence of uneven data distributions across client devices.
We propose em FedClust, a novel approach for CFL that leverages the correlation between local model weights and the data distribution of clients.
arXiv Detail & Related papers (2024-07-09T02:47:16Z) - An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets.
Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round.
We introduce FedAF, a novel aggregation-free FL algorithm.
arXiv Detail & Related papers (2024-04-29T05:55:23Z) - FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm.
It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities.
It achieves substantial and consistent improvements over state-of-the-art baselines.
arXiv Detail & Related papers (2024-02-13T20:04:39Z) - LEFL: Low Entropy Client Sampling in Federated Learning [6.436397118145477]
Federated learning (FL) is a machine learning paradigm where multiple clients collaborate to optimize a single global model using their private data.
We propose LEFL, an alternative sampling strategy by performing a one-time clustering of clients based on their model's learned high-level features.
We show of sampled clients selected with this approach yield a low relative entropy with respect to the global data distribution.
arXiv Detail & Related papers (2023-12-29T01:44:20Z) - PFL-GAN: When Client Heterogeneity Meets Generative Models in
Personalized Federated Learning [55.930403371398114]
We propose a novel generative adversarial network (GAN) sharing and aggregation strategy for personalized learning (PFL)
PFL-GAN addresses the client heterogeneity in different scenarios. More specially, we first learn the similarity among clients and then develop an weighted collaborative data aggregation.
The empirical results through the rigorous experimentation on several well-known datasets demonstrate the effectiveness of PFL-GAN.
arXiv Detail & Related papers (2023-08-23T22:38:35Z) - Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training.
In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework.
Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
arXiv Detail & Related papers (2023-08-11T09:58:47Z) - FedSampling: A Better Sampling Strategy for Federated Learning [81.85411484302952]
Federated learning (FL) is an important technique for learning models from decentralized data in a privacy-preserving way.
Existing FL methods usually uniformly sample clients for local model learning in each round.
We propose a novel data uniform sampling strategy for federated learning (FedSampling)
arXiv Detail & Related papers (2023-06-25T13:38:51Z) - Client Selection in Federated Learning: Principles, Challenges, and
Opportunities [15.33636272844544]
Federated Learning (FL) is a privacy-preserving paradigm for training Machine Learning (ML) models.
In a typical FL scenario, clients exhibit significant heterogeneity in terms of data distribution and hardware configurations.
Various client selection algorithms have been developed, showing promising performance improvement.
arXiv Detail & Related papers (2022-11-03T01:51:14Z) - Straggler-Resilient Personalized Federated Learning [55.54344312542944]
Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions.
We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles.
Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
arXiv Detail & Related papers (2022-06-05T01:14:46Z) - FRAug: Tackling Federated Learning with Non-IID Features via
Representation Augmentation [31.12851987342467]
Federated Learning (FL) is a decentralized learning paradigm in which multiple clients collaboratively train deep learning models.
We propose Federated Representation Augmentation (FRAug) to tackle this practical and challenging problem.
Our approach generates synthetic client-specific samples in the embedding space to augment the usually small client datasets.
arXiv Detail & Related papers (2022-05-30T07:43:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.