Federated Learning with Taskonomy for Non-IID Data
- URL: http://arxiv.org/abs/2103.15947v1
- Date: Mon, 29 Mar 2021 20:47:45 GMT
- Title: Federated Learning with Taskonomy for Non-IID Data
- Authors: Hadi Jamali-Rad, Mohammad Abdizadeh, Attila Szabo
- Abstract summary: We introduce federated learning with taskonomy.
In a one-off process, the server provides the clients with a pretrained (and fine-tunable) encoder to compress their data into a latent representation, and transmit the signature of their data back to the server.
The server then learns the task-relatedness among clients via manifold learning, and performs a generalization of federated averaging.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Classical federated learning approaches incur significant performance
degradation in the presence of non-IID client data. A possible direction to
address this issue is forming clusters of clients with roughly IID data. Most
solutions following this direction are iterative and relatively slow, also
prone to convergence issues in discovering underlying cluster formations. We
introduce federated learning with taskonomy (FLT) that generalizes this
direction by learning the task-relatedness between clients for more efficient
federated aggregation of heterogeneous data. In a one-off process, the server
provides the clients with a pretrained (and fine-tunable) encoder to compress
their data into a latent representation, and transmit the signature of their
data back to the server. The server then learns the task-relatedness among
clients via manifold learning, and performs a generalization of federated
averaging. FLT can flexibly handle a generic client relatedness graph, when
there are no explicit clusters of clients, as well as efficiently decompose it
into (disjoint) clusters for clustered federated learning. We demonstrate that
FLT not only outperforms the existing state-of-the-art baselines in non-IID
scenarios but also offers improved fairness across clients.
Related papers
- A Bayesian Framework for Clustered Federated Learning [14.426129993432193]
One of the main challenges of federated learning (FL) is handling non-independent and identically distributed (non-IID) client data.
We present a unified Bayesian framework for clustered FL which associates clients to clusters.
This work provides insights on client-cluster associations and enables client knowledge sharing in new ways.
arXiv Detail & Related papers (2024-10-20T19:11:24Z) - Fair Federated Data Clustering through Personalization: Bridging the Gap between Diverse Data Distributions [2.7905216619150344]
We introduce the idea of personalization in federated clustering. The goal is achieve balance between achieving lower clustering cost and at same time achieving uniform cost across clients.
We propose p-FClus that addresses these goal in a single round of communication between server and clients.
arXiv Detail & Related papers (2024-07-05T07:10:26Z) - Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training.
In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework.
Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
arXiv Detail & Related papers (2023-08-11T09:58:47Z) - CADIS: Handling Cluster-skewed Non-IID Data in Federated Learning with
Clustered Aggregation and Knowledge DIStilled Regularization [3.3711670942444014]
Federated learning enables edge devices to train a global model collaboratively without exposing their data.
We tackle a new type of Non-IID data, called cluster-skewed non-IID, discovered in actual data sets.
We propose an aggregation scheme that guarantees equality between clusters.
arXiv Detail & Related papers (2023-02-21T02:53:37Z) - Rethinking Data Heterogeneity in Federated Learning: Introducing a New
Notion and Standard Benchmarks [65.34113135080105]
We show that not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants.
Our observations are intuitive.
Our code is available at https://github.com/MMorafah/FL-SC-NIID.
arXiv Detail & Related papers (2022-09-30T17:15:19Z) - Efficient Distribution Similarity Identification in Clustered Federated
Learning via Principal Angles Between Client Data Subspaces [59.33965805898736]
Clustered learning has been shown to produce promising results by grouping clients into clusters.
Existing FL algorithms are essentially trying to group clients together with similar distributions.
Prior FL algorithms attempt similarities indirectly during training.
arXiv Detail & Related papers (2022-09-21T17:37:54Z) - Federated learning with incremental clustering for heterogeneous data [0.0]
In previous approaches, in order to cluster clients the server requires clients to send their parameters simultaneously.
We propose FLIC (Federated Learning with Incremental Clustering) in which the server exploits the updates sent by clients during federated training instead of asking them to send their parameters simultaneously.
We empirically demonstrate for various non-IID cases that our approach successfully splits clients into groups following the same data distributions.
arXiv Detail & Related papers (2022-06-17T13:13:03Z) - On the Convergence of Clustered Federated Learning [57.934295064030636]
In a federated learning system, the clients, e.g. mobile devices and organization participants, usually have different personal preferences or behavior patterns.
This paper proposes a novel weighted client-based clustered FL algorithm to leverage the client's group and each client in a unified optimization framework.
arXiv Detail & Related papers (2022-02-13T02:39:19Z) - Federated Multi-Target Domain Adaptation [99.93375364579484]
Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy.
We consider a more practical scenario where the distributed client data is unlabeled, and a centralized labeled dataset is available on the server.
We propose an effective DualAdapt method to address the new challenges.
arXiv Detail & Related papers (2021-08-17T17:53:05Z) - Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client.
Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation.
This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
arXiv Detail & Related papers (2021-02-14T05:36:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.