Federated Submodel Optimization for Hot and Cold Data Features
- URL: http://arxiv.org/abs/2109.07704v4
- Date: Wed, 5 Apr 2023 12:38:32 GMT
- Title: Federated Submodel Optimization for Hot and Cold Data Features
- Authors: Yucheng Ding, Chaoyue Niu, Fan Wu, Shaojie Tang, Chengfei Lv, Yanghe
Feng, Guihai Chen
- Abstract summary: We study practical data characteristics underlying federated learning, where non-i.i.d. data from clients have sparse features, and a certain client's local data normally involves only a small part of the full model.
Due to data sparsity, the classical federated averaging (FedAvg) algorithm or its variants will be severely slowed down.
We propose federated submodel averaging (FedSubAvg), ensuring that the expectation of the global update of each model parameter is equal to the average of the local updates of the clients.
- Score: 41.99190452773989
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We study practical data characteristics underlying federated learning, where
non-i.i.d. data from clients have sparse features, and a certain client's local
data normally involves only a small part of the full model, called a submodel.
Due to data sparsity, the classical federated averaging (FedAvg) algorithm or
its variants will be severely slowed down, because when updating the global
model, each client's zero update of the full model excluding its submodel is
inaccurately aggregated. Therefore, we propose federated submodel averaging
(FedSubAvg), ensuring that the expectation of the global update of each model
parameter is equal to the average of the local updates of the clients who
involve it. We theoretically proved the convergence rate of FedSubAvg by
deriving an upper bound under a new metric called the element-wise gradient
norm. In particular, this new metric can characterize the convergence of
federated optimization over sparse data, while the conventional metric of
squared gradient norm used in FedAvg and its variants cannot. We extensively
evaluated FedSubAvg over both public and industrial datasets. The evaluation
results demonstrate that FedSubAvg significantly outperforms FedAvg and its
variants.
Related papers
- An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets.
Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round.
We introduce FedAF, a novel aggregation-free FL algorithm.
arXiv Detail & Related papers (2024-04-29T05:55:23Z) - FedImpro: Measuring and Improving Client Update in Federated Learning [77.68805026788836]
Federated Learning (FL) models often experience client drift caused by heterogeneous data.
We present an alternative perspective on client drift and aim to mitigate it by generating improved local models.
arXiv Detail & Related papers (2024-02-10T18:14:57Z) - Federated Learning with Projected Trajectory Regularization [65.6266768678291]
Federated learning enables joint training of machine learning models from distributed clients without sharing their local data.
One key challenge in federated learning is to handle non-identically distributed data across the clients.
We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
arXiv Detail & Related papers (2023-12-22T02:12:08Z) - Federated Learning with Manifold Regularization and Normalized Update
Reaggregation [22.885899072143676]
Federated Learning (FL) is a collaborative machine learning framework where multiple clients train the global model without sharing their own datasets.
In FL, the model inconsistency caused by the local data across clients results in the near-orthogonality of client updates.
We propose FedMRUR by adopting the emerging manifold model fusion scheme and a new global update to alleviate the negative impacts.
arXiv Detail & Related papers (2023-11-10T08:14:27Z) - Federated Learning for Semantic Parsing: Task Formulation, Evaluation
Setup, New Algorithms [29.636944156801327]
Multiple clients collaboratively train one global model without sharing their semantic parsing data.
Lorar adjusts each client's contribution to the global model update based on its training loss reduction during each round.
Clients with smaller datasets enjoy larger performance gains.
arXiv Detail & Related papers (2023-05-26T19:25:49Z) - FedHB: Hierarchical Bayesian Federated Learning [11.936836827864095]
We propose a novel hierarchical Bayesian approach to Federated Learning (FL)
Our model reasonably describes the generative process of clients' local data via hierarchical Bayesian modeling.
We show that our block-coordinate FL algorithm converges to an optimum of the objective at the rate of $O(sqrtt)$.
arXiv Detail & Related papers (2023-05-08T18:21:41Z) - Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients.
FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification.
Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
arXiv Detail & Related papers (2023-05-01T20:04:46Z) - FedAvg with Fine Tuning: Local Updates Lead to Representation Learning [54.65133770989836]
Federated Averaging (FedAvg) algorithm consists of alternating between a few local gradient updates at client nodes, followed by a model averaging update at the server.
We show that the reason behind generalizability of the FedAvg's output is its power in learning the common data representation among the clients' tasks.
We also provide empirical evidence demonstrating FedAvg's representation learning ability in federated image classification with heterogeneous data.
arXiv Detail & Related papers (2022-05-27T00:55:24Z) - Gradient Masked Averaging for Federated Learning [24.687254139644736]
Federated learning allows a large number of clients with heterogeneous data to coordinate learning of a unified global model.
Standard FL algorithms involve averaging of model parameters or gradient updates to approximate the global model at the server.
We propose a gradient masked averaging approach for FL as an alternative to the standard averaging of client updates.
arXiv Detail & Related papers (2022-01-28T08:42:43Z) - Fed-ensemble: Improving Generalization through Model Ensembling in
Federated Learning [5.882234707363695]
Fed-ensemble brings model ensembling to federated learning (FL)
Fed-ensemble can be readily utilized within established FL methods.
arXiv Detail & Related papers (2021-07-21T14:40:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.