An Efficient Framework for Clustered Federated Learning
- URL: http://arxiv.org/abs/2006.04088v2
- Date: Tue, 8 Jun 2021 23:42:59 GMT
- Title: An Efficient Framework for Clustered Federated Learning
- Authors: Avishek Ghosh, Jichan Chung, Dong Yin and Kannan Ramchandran
- Abstract summary: We address the problem of federated learning (FL) where users are distributed into clusters.
We propose the Iterative Federated Clustering Algorithm (IFCA)
We show that our algorithm is efficient in non- partitioned problems such as neural networks.
- Score: 26.24231986590374
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We address the problem of federated learning (FL) where users are distributed
and partitioned into clusters. This setup captures settings where different
groups of users have their own objectives (learning tasks) but by aggregating
their data with others in the same cluster (same learning task), they can
leverage the strength in numbers in order to perform more efficient federated
learning. For this new framework of clustered federated learning, we propose
the Iterative Federated Clustering Algorithm (IFCA), which alternately
estimates the cluster identities of the users and optimizes model parameters
for the user clusters via gradient descent. We analyze the convergence rate of
this algorithm first in a linear model with squared loss and then for generic
strongly convex and smooth loss functions. We show that in both settings, with
good initialization, IFCA is guaranteed to converge, and discuss the optimality
of the statistical error rate. In particular, for the linear model with two
clusters, we can guarantee that our algorithm converges as long as the
initialization is slightly better than random. When the clustering structure is
ambiguous, we propose to train the models by combining IFCA with the weight
sharing technique in multi-task learning. In the experiments, we show that our
algorithm can succeed even if we relax the requirements on initialization with
random initialization and multiple restarts. We also present experimental
results showing that our algorithm is efficient in non-convex problems such as
neural networks. We demonstrate the benefits of IFCA over the baselines on
several clustered FL benchmarks.
Related papers
- Federated cINN Clustering for Accurate Clustered Federated Learning [33.72494731516968]
Federated Learning (FL) presents an innovative approach to privacy-preserving distributed machine learning.
We propose the Federated cINN Clustering Algorithm (FCCA) to robustly cluster clients into different groups.
arXiv Detail & Related papers (2023-09-04T10:47:52Z) - Stochastic Unrolled Federated Learning [85.6993263983062]
We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning.
Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
arXiv Detail & Related papers (2023-05-24T17:26:22Z) - Hard Regularization to Prevent Deep Online Clustering Collapse without
Data Augmentation [65.268245109828]
Online deep clustering refers to the joint use of a feature extraction network and a clustering model to assign cluster labels to each new data point or batch as it is processed.
While faster and more versatile than offline methods, online clustering can easily reach the collapsed solution where the encoder maps all inputs to the same point and all are put into a single cluster.
We propose a method that does not require data augmentation, and that, differently from existing methods, regularizes the hard assignments.
arXiv Detail & Related papers (2023-03-29T08:23:26Z) - Faster Adaptive Federated Learning [84.38913517122619]
Federated learning has attracted increasing attention with the emergence of distributed data.
In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
arXiv Detail & Related papers (2022-12-02T05:07:50Z) - An Improved Algorithm for Clustered Federated Learning [29.166363192740768]
This paper addresses the dichotomy between heterogeneous models and simultaneous training in Federated Learning (FL)
We define a new clustering model for FL based on the (optimal) local models of the users.
textttSR-FCA uses a robust learning algorithm within each cluster to exploit simultaneous training and to correct clustering errors.
arXiv Detail & Related papers (2022-10-20T19:14:36Z) - TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent
Kernels [141.29156234353133]
State-of-the-art convex learning methods can perform far worse than their centralized counterparts when clients have dissimilar data distributions.
We show this disparity can largely be attributed to challenges presented by non-NISTity.
We propose a Train-Convexify neural network (TCT) procedure to sidestep this issue.
arXiv Detail & Related papers (2022-07-13T16:58:22Z) - Personalized Federated Learning via Convex Clustering [72.15857783681658]
We propose a family of algorithms for personalized federated learning with locally convex user costs.
The proposed framework is based on a generalization of convex clustering in which the differences between different users' models are penalized.
arXiv Detail & Related papers (2022-02-01T19:25:31Z) - An Exact Algorithm for Semi-supervised Minimum Sum-of-Squares Clustering [0.5801044612920815]
We present a new branch-and-bound algorithm for semi-supervised MSSC.
Background knowledge is incorporated as pairwise must-link and cannot-link constraints.
For the first time, the proposed global optimization algorithm efficiently manages to solve real-world instances up to 800 data points.
arXiv Detail & Related papers (2021-11-30T17:08:53Z) - Learning Statistical Representation with Joint Deep Embedded Clustering [2.1267423178232407]
StatDEC is an unsupervised framework for joint statistical representation learning and clustering.
Our experiments show that using these representations, one can considerably improve results on imbalanced image clustering across a variety of image datasets.
arXiv Detail & Related papers (2021-09-11T09:26:52Z) - Improving k-Means Clustering Performance with Disentangled Internal
Representations [0.0]
We propose a simpler approach of optimizing the entanglement of the learned latent code representation of an autoencoder.
Using our proposed approach, the test clustering accuracy was 96.2% on the MNIST dataset, 85.6% on the Fashion-MNIST dataset, and 79.2% on the EMNIST Balanced dataset, outperforming our baseline models.
arXiv Detail & Related papers (2020-06-05T11:32:34Z) - FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity
to Non-IID Data [59.50904660420082]
Federated Learning (FL) has become a popular paradigm for learning from distributed data.
To effectively utilize data at different devices without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a "computation then aggregation" (CTA) model.
arXiv Detail & Related papers (2020-05-22T23:07:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.