Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study

• URL: http://arxiv.org/abs/2502.00182v2
• Date: Fri, 07 Feb 2025 14:31:59 GMT
• Title: Understanding Federated Learning from IID to Non-IID dataset: An Experimental Study
• Authors: Jungwon Seo, Ferhat Ozgur Catak, Chunming Rong,
• Abstract summary: federated learning (FL) has emerged as a promising approach for training machine learning models across decentralized data sources without sharing raw data. A significant challenge in FL is that client data are often non-IID (non-independent and identically distributed), leading to reduced performance compared to centralized learning.
• Score: 5.680416078423551
• License:
• Abstract: As privacy concerns and data regulations grow, federated learning (FL) has emerged as a promising approach for training machine learning models across decentralized data sources without sharing raw data. However, a significant challenge in FL is that client data are often non-IID (non-independent and identically distributed), leading to reduced performance compared to centralized learning. While many methods have been proposed to address this issue, their underlying mechanisms are often viewed from different perspectives. Through a comprehensive investigation from gradient descent to FL, and from IID to non-IID data settings, we find that inconsistencies in client loss landscapes primarily cause performance degradation in non-IID scenarios. From this understanding, we observe that existing methods can be grouped into two main strategies: (i) adjusting parameter update paths and (ii) modifying client loss landscapes. These findings offer a clear perspective on addressing non-IID challenges in FL and help guide future research in the field.

Related papers

• Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions [2.9434966603161072]
  Federated Learning (FL) enables users to collectively train ML models without sharing private data. FL struggles when data across clients is not independent and identically distributed (non-IID) data. This technical survey aims to fill that gap by providing a detailed taxonomy for non-IID data, partition protocols, and metrics.
  arXiv  Detail & Related papers  (2024-11-19T09:53:28Z)
• A review on different techniques used to combat the non-IID and heterogeneous nature of data in FL [0.0]
  Federated Learning (FL) is a machine-learning approach enabling collaborative model training across multiple edge devices. The significance of FL is particularly pronounced in industries such as healthcare and finance, where data privacy holds paramount importance. This report delves into the issues arising from non-IID and heterogeneous data and explores current algorithms designed to address these challenges.
  arXiv  Detail & Related papers  (2024-01-01T16:34:00Z)
• Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks [65.34113135080105]
  We show that not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants. Our observations are intuitive. Our code is available at https://github.com/MMorafah/FL-SC-NIID.
  arXiv  Detail & Related papers  (2022-09-30T17:15:19Z)
• Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes [93.61202366677526]
  We study the offline reinforcement learning (RL) in the face of unmeasured confounders. We propose various policy learning methods with the finite-sample suboptimality guarantee of finding the optimal in-class policy.
  arXiv  Detail & Related papers  (2022-09-18T22:03:55Z)
• FEDIC: Federated Learning on Non-IID and Long-Tailed Data via Calibrated Distillation [54.2658887073461]
  Dealing with non-IID data is one of the most challenging problems for federated learning. This paper studies the joint problem of non-IID and long-tailed data in federated learning and proposes a corresponding solution called Federated Ensemble Distillation with Imbalance (FEDIC) FEDIC uses model ensemble to take advantage of the diversity of models trained on non-IID data.
  arXiv  Detail & Related papers  (2022-04-30T06:17:36Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)
• Non-IID data and Continual Learning processes in Federated Learning: A long road ahead [58.720142291102135]
  Federated Learning is a novel framework that allows multiple devices or institutions to train a machine learning model collaboratively while preserving their data private. In this work, we formally classify data statistical heterogeneity and review the most remarkable learning strategies that are able to face it. At the same time, we introduce approaches from other machine learning frameworks, such as Continual Learning, that also deal with data heterogeneity and could be easily adapted to the Federated Learning settings.
  arXiv  Detail & Related papers  (2021-11-26T09:57:11Z)
• Federated Learning on Non-IID Data: A Survey [11.431837357827396]
  Federated learning is an emerging distributed machine learning framework for privacy preservation. Models trained in federated learning usually have worse performance than those trained in the standard centralized learning mode.
  arXiv  Detail & Related papers  (2021-06-12T19:45:35Z)
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data [77.88594632644347]
  Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge. We propose a novel momentum-based method to mitigate this decentralized training difficulty.
  arXiv  Detail & Related papers  (2021-02-09T11:27:14Z)
• Federated Learning on Non-IID Data Silos: An Experimental Study [34.28108345251376]
  Training data have been increasingly fragmented, forming distributed databases of multiple data silos. In this paper, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases.
  arXiv  Detail & Related papers  (2021-02-03T14:29:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.