Global Convergence of Federated Learning for Mixed Regression
- URL: http://arxiv.org/abs/2206.07279v1
- Date: Wed, 15 Jun 2022 03:38:42 GMT
- Title: Global Convergence of Federated Learning for Mixed Regression
- Authors: Lili Su, Jiaming Xu, Pengkun Yang
- Abstract summary: This paper studies the problem of model training under Federated Learning when clients exhibit cluster structure.
Key innovation in our analysis is a uniform estimate on clustering, which we prove by bounding the VC dimension by bounding the general concept classes.
- Score: 17.8469597916875
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper studies the problem of model training under Federated Learning
when clients exhibit cluster structure. We contextualize this problem in mixed
regression, where each client has limited local data generated from one of $k$
unknown regression models. We design an algorithm that achieves global
convergence from any initialization, and works even when local data volume is
highly unbalanced -- there could exist clients that contain $O(1)$ data points
only. Our algorithm first runs moment descent on a few anchor clients (each
with $\tilde{\Omega}(k)$ data points) to obtain coarse model estimates. Then
each client alternately estimates its cluster labels and refines the model
estimates based on FedAvg or FedProx. A key innovation in our analysis is a
uniform estimate on the clustering errors, which we prove by bounding the VC
dimension of general polynomial concept classes based on the theory of
algebraic geometry.
Related papers
- Dual-Criterion Model Aggregation in Federated Learning: Balancing Data Quantity and Quality [0.0]
Federated learning (FL) has become one of the key methods for privacy-preserving collaborative learning.
An aggregation algorithm is recognized as one of the most crucial components for ensuring the efficacy and security of the system.
This study proposes a novel dual-criterion weighted aggregation algorithm involving the quantity and quality of data from the client node.
arXiv Detail & Related papers (2024-11-12T14:09:16Z) - Timely Asynchronous Hierarchical Federated Learning: Age of Convergence [59.96266198512243]
We consider an asynchronous hierarchical federated learning setting with a client-edge-cloud framework.
The clients exchange the trained parameters with their corresponding edge servers, which update the locally aggregated model.
The goal of each client is to converge to the global model, while maintaining timeliness of the clients.
arXiv Detail & Related papers (2023-06-21T17:39:16Z) - Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
Federated learning aims to train models across different clients without the sharing of data for privacy considerations.
We study how data heterogeneity affects the representations of the globally aggregated models.
We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
arXiv Detail & Related papers (2022-10-01T09:04:17Z) - $\texttt{FedBC}$: Calibrating Global and Local Models via Federated
Learning Beyond Consensus [66.62731854746856]
In federated learning (FL), the objective of collaboratively learning a global model through aggregation of model updates across devices tends to oppose the goal of personalization via local information.
In this work, we calibrate this tradeoff in a quantitative manner through a multi-criterion-based optimization.
We demonstrate that $texttFedBC$ balances the global and local model test accuracy metrics across a suite datasets.
arXiv Detail & Related papers (2022-06-22T02:42:04Z) - Robust One Round Federated Learning with Predictive Space Bayesian
Inference [19.533268415744338]
We show how the global predictive posterior can be approximated using client predictive posteriors.
We present an algorithm based on this idea, which performs MCMC sampling at each client to obtain an estimate of the local posterior, and then aggregates these in one round to obtain a global ensemble model.
arXiv Detail & Related papers (2022-06-20T01:06:59Z) - Rate-Distortion Theoretic Bounds on Generalization Error for Distributed
Learning [9.00236182523638]
In this paper, we use tools from rate-distortion theory to establish new upper bounds on the generalization error of statistical distributed learning algorithms.
The bounds depend on the compressibility of each client's algorithm while keeping other clients' algorithms un-compressed.
arXiv Detail & Related papers (2022-06-06T13:21:52Z) - Federated Learning Aggregation: New Robust Algorithms with Guarantees [63.96013144017572]
Federated learning has been recently proposed for distributed model training at the edge.
This paper presents a complete general mathematical convergence analysis to evaluate aggregation strategies in a federated learning framework.
We derive novel aggregation algorithms which are able to modify their model architecture by differentiating client contributions according to the value of their losses.
arXiv Detail & Related papers (2022-05-22T16:37:53Z) - Personalized Federated Learning via Convex Clustering [72.15857783681658]
We propose a family of algorithms for personalized federated learning with locally convex user costs.
The proposed framework is based on a generalization of convex clustering in which the differences between different users' models are penalized.
arXiv Detail & Related papers (2022-02-01T19:25:31Z) - Federated Gaussian Process: Convergence, Automatic Personalization and
Multi-fidelity Modeling [4.18804572788063]
We show that textttFGPR is a promising approach for privacy-preserving multi-fidelity data modeling.
We show that textttFGPR excels in a wide range of applications and is a promising approach for privacy-preserving multi-fidelity data modeling.
arXiv Detail & Related papers (2021-11-28T00:17:31Z) - A Bayesian Federated Learning Framework with Online Laplace
Approximation [144.7345013348257]
Federated learning allows multiple clients to collaboratively learn a globally shared model.
We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side.
We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
arXiv Detail & Related papers (2021-02-03T08:36:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.