FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimator

• URL: http://arxiv.org/abs/2410.03499v1
• Date: Fri, 4 Oct 2024 15:13:31 GMT
• Title: FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimator
• Authors: Sunny Gupta, Nikita Jangid, Amit Sethi,
• Abstract summary: Federated Learning (FL) facilitates data privacy by enabling collaborative in-situ training across decentralized clients. Despite its inherent advantages, FL faces significant challenges of performance and convergence when dealing with data that is not independently and identically distributed. We introduce a novel method designed to address these challenges FedStein: Enhancing Multi-Domain Federated Learning Through the James-Stein Estimator.
• Score: 2.6733991338938026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated Learning (FL) facilitates data privacy by enabling collaborative in-situ training across decentralized clients. Despite its inherent advantages, FL faces significant challenges of performance and convergence when dealing with data that is not independently and identically distributed (non-i.i.d.). While previous research has primarily addressed the issue of skewed label distribution across clients, this study focuses on the less explored challenge of multi-domain FL, where client data originates from distinct domains with varying feature distributions. We introduce a novel method designed to address these challenges FedStein: Enhancing Multi-Domain Federated Learning Through the James-Stein Estimator. FedStein uniquely shares only the James-Stein (JS) estimates of batch normalization (BN) statistics across clients, while maintaining local BN parameters. The non-BN layer parameters are exchanged via standard FL techniques. Extensive experiments conducted across three datasets and multiple models demonstrate that FedStein surpasses existing methods such as FedAvg and FedBN, with accuracy improvements exceeding 14% in certain domains leading to enhanced domain generalization. The code is available at https://github.com/sunnyinAI/FedStein

Related papers

• Optimizing Federated Learning by Entropy-Based Client Selection [13.851391819710367]
  Deep learning domains typically require an extensive amount of data for optimal performance. FedOptEnt is designed to mitigate performance issues caused by label distribution skew. The proposed method outperforms several state-of-the-art algorithms by up to 6% in classification accuracy.
  arXiv  Detail & Related papers  (2024-11-02T13:31:36Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Personalized Federated Learning with Attention-based Client Selection [57.71009302168411]
  We propose FedACS, a new PFL algorithm with an Attention-based Client Selection mechanism. FedACS integrates an attention mechanism to enhance collaboration among clients with similar data distributions. Experiments on CIFAR10 and FMNIST validate FedACS's superiority.
  arXiv  Detail & Related papers  (2023-12-23T03:31:46Z)
• FedWon: Triumphing Multi-domain Federated Learning Without Normalization [50.49210227068574]
  Federated learning (FL) enhances data privacy with collaborative in-situ training on decentralized clients. However, Federated learning (FL) encounters challenges due to non-independent and identically distributed (non-i.i.d) data. We propose a novel method called Federated learning Without normalizations (FedWon) to address the multi-domain problem in FL.
  arXiv  Detail & Related papers  (2023-06-09T13:18:50Z)
• FedFA: Federated Feature Augmentation [25.130087374092383]
  Federated learning allows multiple parties to collaboratively train deep models without exchanging raw data. The primary goal of this paper is to develop a robust federated learning algorithm to address feature shift in clients' samples. We propose FedFA to tackle federated learning from a distinct perspective of federated feature augmentation.
  arXiv  Detail & Related papers  (2023-01-30T15:39:55Z)
• FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation [95.85026305874824]
  We introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency.
  arXiv  Detail & Related papers  (2022-12-14T13:57:01Z)
• FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction [48.85303253333453]
  Federated learning (FL) allows multiple clients to collectively train a high-performance global model without sharing their private data. We propose a novel federated learning algorithm with local drift decoupling and correction (FedDC) Our FedDC only introduces lightweight modifications in the local training phase, in which each client utilizes an auxiliary local drift variable to track the gap between the local model parameter and the global model parameters. Experiment results and analysis demonstrate that FedDC yields expediting convergence and better performance on various image classification tasks.
  arXiv  Detail & Related papers  (2022-03-22T14:06:26Z)
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization [23.519212374186232]
  The emerging paradigm of federated learning (FL) strives to enable collaborative training of deep models on the network edge without centrally aggregating raw data. We propose an effective method that uses local batch normalization to alleviate the feature shift before averaging models. The resulting scheme, called FedBN, outperforms both classical FedAvg and the state-of-the-art for non-iid data.
  arXiv  Detail & Related papers  (2021-02-15T16:04:10Z)
• WAFFLe: Weight Anonymized Factorization for Federated Learning [88.44939168851721]
  In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices. We propose Weight Anonymized Factorization for Federated Learning (WAFFLe), an approach that combines the Indian Buffet Process with a shared dictionary of weight factors for neural networks.
  arXiv  Detail & Related papers  (2020-08-13T04:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.