FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning

• URL: http://arxiv.org/abs/2412.04521v1
• Date: Thu, 05 Dec 2024 12:32:40 GMT
• Title: FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning
• Authors: Jiayu Liu, Yong Wang, Nianbin Wang, Jing Yang, Xiaohui Tao,
• Abstract summary: Federated Learning (FL) is an innovative distributed machine learning paradigm that enables neural network training across devices without centralizing data. Previous research shows that in IID environments, the parameter structure of the model is expected to adhere to certain specific consistency principles. This paper identifies the consistency between the two and leverages it to regulate training, underpinning our proposed FedDW framework. Experimental results show FedDW outperforms 10 state-of-the-art FL methods, improving accuracy by an average of 3% in highly heterogeneous settings.
• Score: 14.477559543490242
• License:
• Abstract: Federated Learning (FL) is an innovative distributed machine learning paradigm that enables neural network training across devices without centralizing data. While this addresses issues of information sharing and data privacy, challenges arise from data heterogeneity across clients and increasing network scale, leading to impacts on model performance and training efficiency. Previous research shows that in IID environments, the parameter structure of the model is expected to adhere to certain specific consistency principles. Thus, identifying and regularizing these consistencies can mitigate issues from heterogeneous data. We found that both soft labels derived from knowledge distillation and the classifier head parameter matrix, when multiplied by their own transpose, capture the intrinsic relationships between data classes. These shared relationships suggest inherent consistency. Therefore, the work in this paper identifies the consistency between the two and leverages it to regulate training, underpinning our proposed FedDW framework. Experimental results show FedDW outperforms 10 state-of-the-art FL methods, improving accuracy by an average of 3% in highly heterogeneous settings. Additionally, we provide a theoretical proof that FedDW offers higher efficiency, with the additional computational load from backpropagation being negligible. The code is available at https://github.com/liuvvvvv1/FedDW.

Related papers

• Learnable Sparse Customization in Heterogeneous Edge Computing [27.201987866208484]
  We propose Learnable Personalized Sparsification for heterogeneous Federated learning (FedLPS) FedLPS learns the importance of model units on local data representation and derives an importance-based sparse pattern with minimals to accurately extract personalized data features. Experiments show that FedLPS outperforms status quo approaches in accuracy and training costs.
  arXiv  Detail & Related papers  (2024-12-10T06:14:31Z)
• One-shot Federated Learning via Synthetic Distiller-Distillate Communication [63.89557765137003]
  One-shot Federated learning (FL) is a powerful technology facilitating collaborative training of machine learning models in a single round of communication. We propose FedSD2C, a novel and practical one-shot FL framework designed to address these challenges.
  arXiv  Detail & Related papers  (2024-12-06T17:05:34Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• FLIGAN: Enhancing Federated Learning with Incomplete Data using GAN [1.5749416770494706]
  Federated Learning (FL) provides a privacy-preserving mechanism for distributed training of machine learning models on networked devices. We propose FLIGAN, a novel approach to address the issue of data incompleteness in FL. Our methodology adheres to FL's privacy requirements by generating synthetic data in a federated manner without sharing the actual data in the process.
  arXiv  Detail & Related papers  (2024-03-25T16:49:38Z)
• FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss for Unlabeled Clients [19.3885479917635]
  Federated learning (FL) is a distributed learning paradigm that facilitates collaborative training of a shared global model across devices. We propose FedAnchor, an innovative FSSL method that introduces a unique double-head structure, called anchor head, paired with the classification head trained exclusively on labeled anchor data on the server. Our approach mitigates the confirmation bias and overfitting issues associated with pseudo-labeling techniques based on high-confidence model prediction samples.
  arXiv  Detail & Related papers  (2024-02-15T18:48:21Z)
• FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
  FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm. It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities. It achieves substantial and consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:04:39Z)
• Mitigating Data Absence in Federated Learning Using Privacy-Controllable Data Digests [12.65383500988952]
  We introduce the Federated Learning with Data Digest (FedDig) framework. FedDig manages unexpected distribution changes using a novel privacy-controllable data digest representation. It consistently outperforms five baseline algorithms by substantial margins in various data absence scenarios.
  arXiv  Detail & Related papers  (2022-10-03T06:54:44Z)
• Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks [65.34113135080105]
  We show that not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants. Our observations are intuitive. Our code is available at https://github.com/MMorafah/FL-SC-NIID.
  arXiv  Detail & Related papers  (2022-09-30T17:15:19Z)
• FedHiSyn: A Hierarchical Synchronous Federated Learning Framework for Resource and Data Heterogeneity [56.82825745165945]
  Federated Learning (FL) enables training a global model without sharing the decentralized raw data stored on multiple devices to protect data privacy. We propose a hierarchical synchronous FL framework, i.e., FedHiSyn, to tackle the problems of straggler effects and outdated models. We evaluate the proposed framework based on MNIST, EMNIST, CIFAR10 and CIFAR100 datasets and diverse heterogeneous settings of devices.
  arXiv  Detail & Related papers  (2022-06-21T17:23:06Z)
• Heterogeneous Federated Learning via Grouped Sequential-to-Parallel Training [60.892342868936865]
  Federated learning (FL) is a rapidly growing privacy-preserving collaborative machine learning paradigm. We propose a data heterogeneous-robust FL approach, FedGSP, to address this challenge. We show that FedGSP improves the accuracy by 3.7% on average compared with seven state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-01-31T03:15:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.