Optimal Client Sampling for Federated Learning
- URL: http://arxiv.org/abs/2010.13723v3
- Date: Mon, 22 Aug 2022 15:38:15 GMT
- Title: Optimal Client Sampling for Federated Learning
- Authors: Wenlin Chen, Samuel Horvath, Peter Richtarik
- Abstract summary: We restrict the number of clients allowed to communicate their updates back to the master node.
In each communication round, all participating clients compute their updates, but only the ones with "important" updates communicate back to the master.
We show that importance can be measured using only the norm of the update and give a formula for optimal client participation.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: It is well understood that client-master communication can be a primary
bottleneck in Federated Learning. In this work, we address this issue with a
novel client subsampling scheme, where we restrict the number of clients
allowed to communicate their updates back to the master node. In each
communication round, all participating clients compute their updates, but only
the ones with "important" updates communicate back to the master. We show that
importance can be measured using only the norm of the update and give a formula
for optimal client participation. This formula minimizes the distance between
the full update, where all clients participate, and our limited update, where
the number of participating clients is restricted. In addition, we provide a
simple algorithm that approximates the optimal formula for client
participation, which only requires secure aggregation and thus does not
compromise client privacy. We show both theoretically and empirically that for
Distributed SGD (DSGD) and Federated Averaging (FedAvg), the performance of our
approach can be close to full participation and superior to the baseline where
participating clients are sampled uniformly. Moreover, our approach is
orthogonal to and compatible with existing methods for reducing communication
overhead, such as local methods and communication compression methods.
Related papers
- Personalized Federated Knowledge Graph Embedding with Client-Wise Relation Graph [49.66272783945571]
We propose Personalized Federated knowledge graph Embedding with client-wise relation graph.
PFedEG learns personalized supplementary knowledge for each client by amalgamating entity embedding from its neighboring clients.
We conduct extensive experiments on four benchmark datasets to evaluate our method against state-of-the-art models.
arXiv Detail & Related papers (2024-06-17T17:44:53Z) - Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning [51.560590617691005]
We investigate whether it is possible to squeeze more juice" out of each cohort than what is possible in a single communication round.
Our approach leads to up to 74% reduction in the total communication cost needed to train a FL model in the cross-device setting.
arXiv Detail & Related papers (2024-06-03T08:48:49Z) - Emulating Full Client Participation: A Long-Term Client Selection Strategy for Federated Learning [48.94952630292219]
We propose a novel client selection strategy designed to emulate the performance achieved with full client participation.
In a single round, we select clients by minimizing the gradient-space estimation error between the client subset and the full client set.
In multi-round selection, we introduce a novel individual fairness constraint, which ensures that clients with similar data distributions have similar frequencies of being selected.
arXiv Detail & Related papers (2024-05-22T12:27:24Z) - Utilizing Free Clients in Federated Learning for Focused Model
Enhancement [9.370655190768163]
Federated Learning (FL) is a distributed machine learning approach to learn models on decentralized heterogeneous data.
We present FedALIGN (Federated Adaptive Learning with Inclusion of Global Needs) to address this challenge.
arXiv Detail & Related papers (2023-10-06T18:23:40Z) - Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated
Learning [14.196701066823499]
In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes.
We show that individual client models experience a catastrophic forgetting with respect to data from other clients.
We propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss.
arXiv Detail & Related papers (2023-04-11T14:51:55Z) - FilFL: Client Filtering for Optimized Client Participation in Federated Learning [71.46173076298957]
Federated learning enables clients to collaboratively train a model without exchanging local data.
Clients participating in the training process significantly impact the convergence rate, learning efficiency, and model generalization.
We propose a novel approach, client filtering, to improve model generalization and optimize client participation and training.
arXiv Detail & Related papers (2023-02-13T18:55:31Z) - Federated Learning with Regularized Client Participation [1.433758865948252]
Federated Learning (FL) is a distributed machine learning approach where multiple clients work together to solve a machine learning task.
One of the key challenges in FL is the issue of partial participation, which occurs when a large number of clients are involved in the training process.
We propose a new technique and design a novel regularized client participation scheme.
arXiv Detail & Related papers (2023-02-07T18:26:07Z) - Communication-Efficient Federated Learning with Accelerated Client Gradient [46.81082897703729]
Federated learning often suffers from slow and unstable convergence due to the heterogeneous characteristics of participating client datasets.
We propose a simple but effective federated learning framework, which improves the consistency across clients and facilitates the convergence of the server model.
We provide the theoretical convergence rate of our algorithm and demonstrate remarkable performance gains in terms of accuracy and communication efficiency.
arXiv Detail & Related papers (2022-01-10T05:31:07Z) - Federated Multi-Target Domain Adaptation [99.93375364579484]
Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy.
We consider a more practical scenario where the distributed client data is unlabeled, and a centralized labeled dataset is available on the server.
We propose an effective DualAdapt method to address the new challenges.
arXiv Detail & Related papers (2021-08-17T17:53:05Z) - Shuffled Model of Federated Learning: Privacy, Communication and
Accuracy Trade-offs [30.58690911428577]
We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements.
We develop (optimal) communication-efficient schemes for private mean estimation for several $ell_p$ spaces.
We demonstrate that one can get the same privacy, optimization-performance operating point developed in recent methods that use full-precision communication.
arXiv Detail & Related papers (2020-08-17T09:41:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.