Prototype Helps Federated Learning: Towards Faster Convergence
- URL: http://arxiv.org/abs/2303.12296v1
- Date: Wed, 22 Mar 2023 04:06:29 GMT
- Title: Prototype Helps Federated Learning: Towards Faster Convergence
- Authors: Yu Qiao, Seong-Bae Park, Sun Moo Kang, and Choong Seon Hong
- Abstract summary: Federated learning (FL) is a distributed machine learning technique in which multiple clients cooperate to train a shared model without exchanging their raw data.
In this paper, a prototype-based federated learning framework is proposed, which can achieve better inference performance with only a few changes to the last global iteration of the typical federated learning process.
- Score: 38.517903009319994
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Federated learning (FL) is a distributed machine learning technique in which
multiple clients cooperate to train a shared model without exchanging their raw
data. However, heterogeneity of data distribution among clients usually leads
to poor model inference. In this paper, a prototype-based federated learning
framework is proposed, which can achieve better inference performance with only
a few changes to the last global iteration of the typical federated learning
process. In the last iteration, the server aggregates the prototypes
transmitted from distributed clients and then sends them back to local clients
for their respective model inferences. Experiments on two baseline datasets
show that our proposal can achieve higher accuracy (at least 1%) and relatively
efficient communication than two popular baselines under different
heterogeneous settings.
Related papers
- Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains [8.047147770476212]
Federated learning (FL) allows collaborative machine learning training without sharing private data.
While most FL methods assume identical data domains across clients, real-world scenarios often involve heterogeneous data domains.
We introduce FedPLVM, which establishes variance-aware dual-level prototypes clustering and employs a novel $alpha$-sparsity prototype loss.
arXiv Detail & Related papers (2024-03-14T02:36:16Z) - Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training.
In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework.
Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
arXiv Detail & Related papers (2023-08-11T09:58:47Z) - Boosting Federated Learning Convergence with Prototype Regularization [46.23872011375391]
This paper introduces a prototype-based regularization strategy to address the heterogeneous data distribution among clients.
The experimental results on MNIST and Fashion-MNIST show that our proposal achieves improvements of 3.3% and 8.9% in average test accuracy.
arXiv Detail & Related papers (2023-07-20T04:35:50Z) - Optimizing Server-side Aggregation For Robust Federated Learning via
Subspace Training [80.03567604524268]
Non-IID data distribution across clients and poisoning attacks are two main challenges in real-world federated learning systems.
We propose SmartFL, a generic approach that optimize the server-side aggregation process.
We provide theoretical analyses of the convergence and generalization capacity for SmartFL.
arXiv Detail & Related papers (2022-11-10T13:20:56Z) - Efficient Distribution Similarity Identification in Clustered Federated
Learning via Principal Angles Between Client Data Subspaces [59.33965805898736]
Clustered learning has been shown to produce promising results by grouping clients into clusters.
Existing FL algorithms are essentially trying to group clients together with similar distributions.
Prior FL algorithms attempt similarities indirectly during training.
arXiv Detail & Related papers (2022-09-21T17:37:54Z) - Cross-domain Federated Object Detection [43.66352018668227]
Federated learning can enable multi-party collaborative learning without leaking client data.
We propose a cross-domain federated object detection framework, named FedOD.
arXiv Detail & Related papers (2022-06-30T03:09:59Z) - FedAvg with Fine Tuning: Local Updates Lead to Representation Learning [54.65133770989836]
Federated Averaging (FedAvg) algorithm consists of alternating between a few local gradient updates at client nodes, followed by a model averaging update at the server.
We show that the reason behind generalizability of the FedAvg's output is its power in learning the common data representation among the clients' tasks.
We also provide empirical evidence demonstrating FedAvg's representation learning ability in federated image classification with heterogeneous data.
arXiv Detail & Related papers (2022-05-27T00:55:24Z) - Personalized Federated Learning by Structured and Unstructured Pruning
under Data Heterogeneity [3.291862617649511]
We propose a new approach for obtaining a personalized model from a client-level objective.
To realize this personalization, we leverage finding a small subnetwork for each client.
arXiv Detail & Related papers (2021-05-02T22:10:46Z) - A Bayesian Federated Learning Framework with Online Laplace
Approximation [144.7345013348257]
Federated learning allows multiple clients to collaboratively learn a globally shared model.
We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side.
We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
arXiv Detail & Related papers (2021-02-03T08:36:58Z) - Multi-Center Federated Learning [62.57229809407692]
This paper proposes a novel multi-center aggregation mechanism for federated learning.
It learns multiple global models from the non-IID user data and simultaneously derives the optimal matching between users and centers.
Our experimental results on benchmark datasets show that our method outperforms several popular federated learning methods.
arXiv Detail & Related papers (2020-05-03T09:14:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.