Rethinking Client Drift in Federated Learning: A Logit Perspective
- URL: http://arxiv.org/abs/2308.10162v1
- Date: Sun, 20 Aug 2023 04:41:01 GMT
- Title: Rethinking Client Drift in Federated Learning: A Logit Perspective
- Authors: Yunlu Yan, Chun-Mei Feng, Mang Ye, Wangmeng Zuo, Ping Li, Rick Siow
Mong Goh, Lei Zhu, C. L. Philip Chen
- Abstract summary: Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection.
We find that the difference in logits between the local and global models increases as the model is continuously updated.
We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
- Score: 125.35844582366441
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated Learning (FL) enables multiple clients to collaboratively learn in
a distributed way, allowing for privacy protection. However, the real-world
non-IID data will lead to client drift which degrades the performance of FL.
Interestingly, we find that the difference in logits between the local and
global models increases as the model is continuously updated, thus seriously
deteriorating FL performance. This is mainly due to catastrophic forgetting
caused by data heterogeneity between clients. To alleviate this problem, we
propose a new algorithm, named FedCSD, a Class prototype Similarity
Distillation in a federated framework to align the local and global models.
FedCSD does not simply transfer global knowledge to local clients, as an
undertrained global model cannot provide reliable knowledge, i.e., class
similarity information, and its wrong soft labels will mislead the optimization
of local models. Concretely, FedCSD introduces a class prototype similarity
distillation to align the local logits with the refined global logits that are
weighted by the similarity between local logits and the global prototype. To
enhance the quality of global logits, FedCSD adopts an adaptive mask to filter
out the terrible soft labels of the global models, thereby preventing them to
mislead local optimization. Extensive experiments demonstrate the superiority
of our method over the state-of-the-art federated learning approaches in
various heterogeneous settings. The source code will be released.
Related papers
- FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning [27.782676760198697]
Federated Learning (FL) has emerged as a pivotal framework for the development of effective global models.
A key challenge in FL is client drift, where data heterogeneity impedes the aggregation of scattered knowledge.
We introduce a novel algorithm named FedDr+, which empowers local model alignment using dot-regression loss.
arXiv Detail & Related papers (2024-06-04T14:34:13Z) - An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets.
Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round.
We introduce FedAF, a novel aggregation-free FL algorithm.
arXiv Detail & Related papers (2024-04-29T05:55:23Z) - Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources.
The lack of model privacy protection in FL becomes an unneglectable challenge.
We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
arXiv Detail & Related papers (2023-11-12T11:01:10Z) - Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training.
In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework.
Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
arXiv Detail & Related papers (2023-08-11T09:58:47Z) - FedSoup: Improving Generalization and Personalization in Federated
Learning via Selective Model Interpolation [32.36334319329364]
Cross-silo federated learning (FL) enables the development of machine learning models on datasets distributed across data centers.
Recent research has found that current FL algorithms face a trade-off between local and global performance when confronted with distribution shifts.
We propose a novel federated model soup method to optimize the trade-off between local and global performance.
arXiv Detail & Related papers (2023-07-20T00:07:29Z) - FedCME: Client Matching and Classifier Exchanging to Handle Data
Heterogeneity in Federated Learning [5.21877373352943]
Data heterogeneity across clients is one of the key challenges in Federated Learning (FL)
We propose a novel FL framework named FedCME by client matching and classifier exchanging.
Experimental results demonstrate that FedCME performs better than FedAvg, FedProx, MOON and FedRS on popular federated learning benchmarks.
arXiv Detail & Related papers (2023-07-17T15:40:45Z) - No One Left Behind: Real-World Federated Class-Incremental Learning [111.77681016996202]
Local-Global Anti-forgetting (LGA) model addresses local and global catastrophic forgetting.
We develop a category-balanced gradient-adaptive compensation loss and a category gradient-induced semantic distillation loss.
It augments perturbed prototype images of new categories collected from local clients via self-supervised prototype augmentation.
arXiv Detail & Related papers (2023-02-02T06:41:02Z) - Fine-tuning Global Model via Data-Free Knowledge Distillation for
Non-IID Federated Learning [86.59588262014456]
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint.
We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG)
Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
arXiv Detail & Related papers (2022-03-17T11:18:17Z) - Preservation of the Global Knowledge by Not-True Self Knowledge
Distillation in Federated Learning [8.474470736998136]
In Federated Learning (FL), a strong global model is collaboratively learned by aggregating the clients' locally trained models.
We observe that fitting on biased local distribution shifts the feature on global distribution and results in forgetting of global knowledge.
We propose a simple yet effective framework Federated Local Self-Distillation (FedLSD), which utilizes the global knowledge on locally available data.
arXiv Detail & Related papers (2021-06-06T11:51:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.