Clustered Federated Learning via Generalized Total Variation
Minimization
- URL: http://arxiv.org/abs/2105.12769v4
- Date: Sun, 18 Jun 2023 17:14:37 GMT
- Title: Clustered Federated Learning via Generalized Total Variation
Minimization
- Authors: Yasmin SarcheshmehPour, Yu Tian, Linli Zhang, Alexander Jung
- Abstract summary: We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure.
Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV)
Our main algorithmic contribution is a fully decentralized federated learning algorithm.
- Score: 83.26141667853057
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We study optimization methods to train local (or personalized) models for
decentralized collections of local datasets with an intrinsic network
structure. This network structure arises from domain-specific notions of
similarity between local datasets. Examples for such notions include
spatio-temporal proximity, statistical dependencies or functional relations.
Our main conceptual contribution is to formulate federated learning as
generalized total variation (GTV) minimization. This formulation unifies and
considerably extends existing federated learning methods. It is highly flexible
and can be combined with a broad range of parametric models, including
generalized linear models or deep neural networks. Our main algorithmic
contribution is a fully decentralized federated learning algorithm. This
algorithm is obtained by applying an established primal-dual method to solve
GTV minimization. It can be implemented as message passing and is robust
against inexact computations that arise from limited computational resources
including processing time or bandwidth. Our main analytic contribution is an
upper bound on the deviation between the local model parameters learnt by our
algorithm and an oracle-based clustered federated learning method. This upper
bound reveals conditions on the local models and the network structure of local
datasets such that GTV minimization is able to pool (nearly) homogeneous local
datasets.
Related papers
- Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data.
Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
arXiv Detail & Related papers (2023-07-12T16:28:21Z) - Federated Gradient Matching Pursuit [17.695717854068715]
Traditional machine learning techniques require centralizing all training data on one server or data hub.
In particular, federated learning (FL) provides such a solution to learn a shared model while keeping training data at local clients.
We propose a novel algorithmic framework, federated gradient matching pursuit (FedGradMP), to solve the sparsity constrained minimization problem in the FL setting.
arXiv Detail & Related papers (2023-02-20T16:26:29Z) - Towards Model-Agnostic Federated Learning over Networks [0.0]
We present a model-agnostic federated learning method for networks of heterogeneous data and models.
Our method is an instance of empirical risk minimization, with the regularization term derived from the network structure of data.
arXiv Detail & Related papers (2023-02-08T22:55:57Z) - FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for
Federated Learning on Non-IID Data [69.0785021613868]
Federated learning is a distributed machine learning approach which enables a shared server model to learn by aggregating the locally-computed parameter updates with the training data from spatially-distributed client silos.
We propose the Federated Invariant Learning Consistency (FedILC) approach, which leverages the gradient covariance and the geometric mean of Hessians to capture both inter-silo and intra-silo consistencies.
This is relevant to various fields such as medical healthcare, computer vision, and the Internet of Things (IoT)
arXiv Detail & Related papers (2022-05-19T03:32:03Z) - Global Aggregation then Local Distribution for Scene Parsing [99.1095068574454]
We show that our approach can be modularized as an end-to-end trainable block and easily plugged into existing semantic segmentation networks.
Our approach allows us to build new state of the art on major semantic segmentation benchmarks including Cityscapes, ADE20K, Pascal Context, Camvid and COCO-stuff.
arXiv Detail & Related papers (2021-07-28T03:46:57Z) - Edge-assisted Democratized Learning Towards Federated Analytics [67.44078999945722]
We show the hierarchical learning structure of the proposed edge-assisted democratized learning mechanism, namely Edge-DemLearn.
We also validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions.
arXiv Detail & Related papers (2020-12-01T11:46:03Z) - FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity
to Non-IID Data [59.50904660420082]
Federated Learning (FL) has become a popular paradigm for learning from distributed data.
To effectively utilize data at different devices without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a "computation then aggregation" (CTA) model.
arXiv Detail & Related papers (2020-05-22T23:07:42Z) - Fast local linear regression with anchor regularization [21.739281173516247]
We propose a simple yet effective local model training algorithm called the fast anchor regularized local linear method (FALL)
Through experiments on synthetic and real-world datasets, we demonstrate that FALL compares favorably in terms of accuracy with the state-of-the-art network Lasso algorithm.
arXiv Detail & Related papers (2020-02-21T10:03:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.