Communication-Efficient Agnostic Federated Averaging
- URL: http://arxiv.org/abs/2104.02748v1
- Date: Tue, 6 Apr 2021 19:01:18 GMT
- Title: Communication-Efficient Agnostic Federated Averaging
- Authors: Jae Ro, Mingqing Chen, Rajiv Mathews, Mehryar Mohri, Ananda Theertha
Suresh
- Abstract summary: In distributed learning settings, the training algorithm can be potentially biased towards different clients.
We propose a communication-efficient distributed algorithm called Agnostic Federated Averaging (or AgnosticFedAvg) to minimize the domain-agnostic objective proposed in Mohri et al.
- Score: 39.761808414613185
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In distributed learning settings such as federated learning, the training
algorithm can be potentially biased towards different clients. Mohri et al.
(2019) proposed a domain-agnostic learning algorithm, where the model is
optimized for any target distribution formed by a mixture of the client
distributions in order to overcome this bias. They further proposed an
algorithm for the cross-silo federated learning setting, where the number of
clients is small. We consider this problem in the cross-device setting, where
the number of clients is much larger. We propose a communication-efficient
distributed algorithm called Agnostic Federated Averaging (or AgnosticFedAvg)
to minimize the domain-agnostic objective proposed in Mohri et al. (2019),
which is amenable to other private mechanisms such as secure aggregation. We
highlight two types of naturally occurring domains in federated learning and
argue that AgnosticFedAvg performs well on both. To demonstrate the practical
effectiveness of AgnosticFedAvg, we report positive results for large-scale
language modeling tasks in both simulation and live experiments, where the
latter involves training language models for Spanish virtual keyboard for
millions of user devices.
Related papers
- SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learning [48.072207894076556]
Cross-device training is a subfield of learning where the number of clients can reach into the billions.
Standard approaches and local methods are prone to issues as crucial as cross-device similarity.
Our method is the first in its kind, that does not require the objective and provably benefits from clients having similar data.
arXiv Detail & Related papers (2024-05-30T15:07:30Z) - Learn What You Need in Personalized Federated Learning [53.83081622573734]
$textitLearn2pFed$ is a novel algorithm-unrolling-based personalized federated learning framework.
We show that $textitLearn2pFed$ significantly outperforms previous personalized federated learning methods.
arXiv Detail & Related papers (2024-01-16T12:45:15Z) - Effectively Heterogeneous Federated Learning: A Pairing and Split
Learning Based Approach [16.093068118849246]
This paper presents a novel split federated learning (SFL) framework that pairs clients with different computational resources.
A greedy algorithm is proposed by reconstructing the optimization of training latency as a graph edge selection problem.
Simulation results show the proposed method can significantly improve the FL training speed and achieve high performance.
arXiv Detail & Related papers (2023-08-26T11:10:54Z) - Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data.
Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
arXiv Detail & Related papers (2023-07-12T16:28:21Z) - Faster Adaptive Federated Learning [84.38913517122619]
Federated learning has attracted increasing attention with the emergence of distributed data.
In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
arXiv Detail & Related papers (2022-12-02T05:07:50Z) - FedPop: A Bayesian Approach for Personalised Federated Learning [25.67466138369391]
Personalised federated learning aims at collaboratively learning a machine learning model taylored for each client.
We propose a novel methodology coined FedPop by recasting personalised FL into the population modeling paradigm.
Compared to existing personalised FL methods, the proposed methodology has important benefits: it is robust to client drift, practical for inference on new clients, and above all, enables uncertainty quantification under mild computational and memory overheads.
arXiv Detail & Related papers (2022-06-07T22:52:59Z) - DRFLM: Distributionally Robust Federated Learning with Inter-client
Noise via Local Mixup [58.894901088797376]
federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data.
We propose a general framework to solve the above two challenges simultaneously.
We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
arXiv Detail & Related papers (2022-04-16T08:08:29Z) - Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains.
We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
arXiv Detail & Related papers (2021-12-12T06:11:16Z) - A Federated Learning Aggregation Algorithm for Pervasive Computing:
Evaluation and Comparison [0.6299766708197883]
Pervasive computing promotes the installation of connected devices in our living spaces in order to provide services.
Two major developments have gained significant momentum recently: an advanced use of edge resources and the integration of machine learning techniques for engineering applications.
We propose a novel aggregation algorithm, termed FedDist, which is able to modify its model architecture by identifying dissimilarities between specific neurons amongst the clients.
arXiv Detail & Related papers (2021-10-19T19:43:28Z) - Federated Multi-Task Learning under a Mixture of Distributions [10.00087964926414]
Federated Learning (FL) is a framework for on-device collaborative training of machine learning models.
First efforts in FL focused on learning a single global model with good average performance across clients, but the global model may be arbitrarily bad for a given client.
We study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions.
arXiv Detail & Related papers (2021-08-23T15:47:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.