Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization

• URL: http://arxiv.org/abs/2309.06692v2
• Date: Thu, 7 Mar 2024 14:57:19 GMT
• Title: Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization
• Authors: Xinyu Zhang, Weiyu Sun, Ying Chen
• Abstract summary: Federated learning (FL) is a privacy-preserving paradigm for collaboratively training a global model from decentralized clients. In this work, we revisit this key challenge through the lens of gradient conflicts on the server side. We propose FedGH, a simple yet effective method that mitigates local drifts through Gradient Harmonization.
• Score: 11.484136481586381
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) is a privacy-preserving paradigm for collaboratively training a global model from decentralized clients. However, the performance of FL is hindered by non-independent and identically distributed (non-IID) data and device heterogeneity. In this work, we revisit this key challenge through the lens of gradient conflicts on the server side. Specifically, we first investigate the gradient conflict phenomenon among multiple clients and reveal that stronger heterogeneity leads to more severe gradient conflicts. To tackle this issue, we propose FedGH, a simple yet effective method that mitigates local drifts through Gradient Harmonization. This technique projects one gradient vector onto the orthogonal plane of the other within conflicting client pairs. Extensive experiments demonstrate that FedGH consistently enhances multiple state-of-the-art FL baselines across diverse benchmarks and non-IID scenarios. Notably, FedGH yields more significant improvements in scenarios with stronger heterogeneity. As a plug-and-play module, FedGH can be seamlessly integrated into any FL framework without requiring hyperparameter tuning.

Related papers

• FedSKD: Aggregation-free Model-heterogeneous Federated Learning using Multi-dimensional Similarity Knowledge Distillation [7.944298319589845]
  Federated learning (FL) enables privacy-preserving collaborative model training without direct data sharing. Model-heterogeneous FL (MHFL) allows clients to train personalized models with heterogeneous architectures tailored to their computational resources and application-specific needs. While peer-to-peer (P2P) FL removes server dependence, it suffers from model drift and knowledge dilution, limiting its effectiveness in heterogeneous settings. We propose FedSKD, a novel MHFL framework that facilitates direct knowledge exchange through round-robin model circulation.
  arXiv  Detail & Related papers  (2025-03-23T05:33:10Z)
• A Unified Solution to Diverse Heterogeneities in One-shot Federated Learning [16.120630244663452]
  One-Shot Federated Learning (OSFL) restricts communication between the server and clients to a single round. FedHydra is a unified, data-free, OSFL framework designed to effectively address both model and data heterogeneity.
  arXiv  Detail & Related papers  (2024-10-28T15:20:52Z)
• Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradients [11.269920973751244]
  In personalized Federated Learning (pFL), high data heterogeneity can cause significant gradient divergence across devices. We introduce a new approach to pFL design, namely Federated Learning with Layer-wise Aggregation via Gradient Analysis (FedLAG) FedLAG assigns layers for personalization based on the extent of layer-wise gradient conflicts.
  arXiv  Detail & Related papers  (2024-10-03T14:46:19Z)
• Hierarchical Federated Learning with Multi-Timescale Gradient Correction [24.713834338757195]
  In this paper, we propose a multi-time correction (MTGC) methodology to resolve this issue. Our key idea is to introduce distinct control to (i) correct the client gradient the group gradient, i.e., to reduce client model drift caused by local updates based on individual datasets.
  arXiv  Detail & Related papers  (2024-09-27T05:10:05Z)
• FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
  FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm. It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities. It achieves substantial and consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:04:39Z)
• FedImpro: Measuring and Improving Client Update in Federated Learning [77.68805026788836]
  Federated Learning (FL) models often experience client drift caused by heterogeneous data. We present an alternative perspective on client drift and aim to mitigate it by generating improved local models.
  arXiv  Detail & Related papers  (2024-02-10T18:14:57Z)
• Take History as a Mirror in Heterogeneous Federated Learning [9.187993085263209]
  Federated Learning (FL) allows several clients to cooperatively train machine learning models without disclosing the raw data. In this work, we propose a novel asynchronous FL framework called Federated Historical Learning (FedHist) FedHist effectively addresses the challenges posed by both Non-IID data and gradient staleness.
  arXiv  Detail & Related papers  (2023-12-16T11:40:49Z)
• Enabling Quartile-based Estimated-Mean Gradient Aggregation As Baseline for Federated Image Classifications [5.5099914877576985]
  Federated Learning (FL) has revolutionized how we train deep neural networks by enabling decentralized collaboration while safeguarding sensitive data and improving model performance. This paper introduces an innovative solution named Estimated Mean Aggregation (EMA) that not only addresses these challenges but also provides a fundamental reference point as a $mathsfbaseline$ for advanced aggregation techniques in FL systems.
  arXiv  Detail & Related papers  (2023-09-21T17:17:28Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)
• FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering [4.489171618387544]
  Federated Learning (FL) is a machine learning paradigm that safeguards privacy by retaining client data on edge devices. In this paper, we identify the learning challenges posed by the simultaneous occurrence of diverse distribution shifts. We propose a novel clustering algorithm framework, dubbed as FedRC, which adheres to our proposed clustering principle.
  arXiv  Detail & Related papers  (2023-01-29T06:50:45Z)
• Beyond ADMM: A Unified Client-variance-reduced Adaptive Federated Learning Framework [82.36466358313025]
  We propose a primal-dual FL algorithm, termed FedVRA, that allows one to adaptively control the variance-reduction level and biasness of the global model. Experiments based on (semi-supervised) image classification tasks demonstrate superiority of FedVRA over the existing schemes.
  arXiv  Detail & Related papers  (2022-12-03T03:27:51Z)
• FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning [91.74206675452888]
  We propose a novel method FedFM, which guides each client's features to match shared category-wise anchors. To achieve higher efficiency and flexibility, we propose a FedFM variant, called FedFM-Lite, where clients communicate with server with fewer synchronization times and communication bandwidth costs.
  arXiv  Detail & Related papers  (2022-10-14T08:11:34Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.