FedWon: Triumphing Multi-domain Federated Learning Without Normalization

• URL: http://arxiv.org/abs/2306.05879v2
• Date: Fri, 26 Jan 2024 05:58:33 GMT
• Title: FedWon: Triumphing Multi-domain Federated Learning Without Normalization
• Authors: Weiming Zhuang, Lingjuan Lyu
• Abstract summary: Federated learning (FL) enhances data privacy with collaborative in-situ training on decentralized clients. However, Federated learning (FL) encounters challenges due to non-independent and identically distributed (non-i.i.d) data. We propose a novel method called Federated learning Without normalizations (FedWon) to address the multi-domain problem in FL.
• Score: 50.49210227068574
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) enhances data privacy with collaborative in-situ training on decentralized clients. Nevertheless, FL encounters challenges due to non-independent and identically distributed (non-i.i.d) data, leading to potential performance degradation and hindered convergence. While prior studies predominantly addressed the issue of skewed label distribution, our research addresses a crucial yet frequently overlooked problem known as multi-domain FL. In this scenario, clients' data originate from diverse domains with distinct feature distributions, instead of label distributions. To address the multi-domain problem in FL, we propose a novel method called Federated learning Without normalizations (FedWon). FedWon draws inspiration from the observation that batch normalization (BN) faces challenges in effectively modeling the statistics of multiple domains, while existing normalization techniques possess their own limitations. In order to address these issues, FedWon eliminates the normalization layers in FL and reparameterizes convolution layers with scaled weight standardization. Through extensive experimentation on five datasets and five models, our comprehensive experimental results demonstrate that FedWon surpasses both FedAvg and the current state-of-the-art method (FedBN) across all experimental setups, achieving notable accuracy improvements of more than 10% in certain domains. Furthermore, FedWon is versatile for both cross-silo and cross-device FL, exhibiting robust domain generalization capability, showcasing strong performance even with a batch size as small as 1, thereby catering to resource-constrained devices. Additionally, FedWon can also effectively tackle the challenge of skewed label distribution.

Related papers

• FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimator [2.6733991338938026]
  Federated Learning (FL) facilitates data privacy by enabling collaborative in-situ training across decentralized clients. Despite its inherent advantages, FL faces significant challenges of performance and convergence when dealing with data that is not independently and identically distributed. We introduce a novel method designed to address these challenges FedStein: Enhancing Multi-Domain Federated Learning Through the James-Stein Estimator.
  arXiv  Detail & Related papers  (2024-10-04T15:13:31Z)
• Beyond ADMM: A Unified Client-variance-reduced Adaptive Federated Learning Framework [82.36466358313025]
  We propose a primal-dual FL algorithm, termed FedVRA, that allows one to adaptively control the variance-reduction level and biasness of the global model. Experiments based on (semi-supervised) image classification tasks demonstrate superiority of FedVRA over the existing schemes.
  arXiv  Detail & Related papers  (2022-12-03T03:27:51Z)
• FL Games: A Federated Learning Framework for Distribution Shifts [71.98708418753786]
  Federated learning aims to train predictive models for data that is distributed across clients, under the orchestration of a server. We propose FL GAMES, a game-theoretic framework for federated learning that learns causal features that are invariant across clients.
  arXiv  Detail & Related papers  (2022-10-31T22:59:03Z)
• FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning [91.74206675452888]
  We propose a novel method FedFM, which guides each client's features to match shared category-wise anchors. To achieve higher efficiency and flexibility, we propose a FedFM variant, called FedFM-Lite, where clients communicate with server with fewer synchronization times and communication bandwidth costs.
  arXiv  Detail & Related papers  (2022-10-14T08:11:34Z)
• Mitigating Data Absence in Federated Learning Using Privacy-Controllable Data Digests [12.65383500988952]
  We introduce the Federated Learning with Data Digest (FedDig) framework. FedDig manages unexpected distribution changes using a novel privacy-controllable data digest representation. It consistently outperforms five baseline algorithms by substantial margins in various data absence scenarios.
  arXiv  Detail & Related papers  (2022-10-03T06:54:44Z)
• FedDAR: Federated Domain-Aware Representation Learning [14.174833360938806]
  Cross-silo Federated learning (FL) has become a promising tool in machine learning applications for healthcare. We propose a novel method, FedDAR, which learns a domain shared representation and domain-wise personalized prediction heads.
  arXiv  Detail & Related papers  (2022-09-08T19:18:59Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Towards Understanding Quality Challenges of the Federated Learning: A First Look from the Lens of Robustness [4.822471415125479]
  Federated learning (FL) aims to preserve users' data privacy while leveraging the entire dataset of all participants for training. FL still tends to suffer from quality issues such as attacks or byzantine faults. This paper investigates the effectiveness of state-of-the-art (SOTA) robust FL techniques in the presence of attacks and faults.
  arXiv  Detail & Related papers  (2022-01-05T02:06:39Z)
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization [23.519212374186232]
  The emerging paradigm of federated learning (FL) strives to enable collaborative training of deep models on the network edge without centrally aggregating raw data. We propose an effective method that uses local batch normalization to alleviate the feature shift before averaging models. The resulting scheme, called FedBN, outperforms both classical FedAvg and the state-of-the-art for non-iid data.
  arXiv  Detail & Related papers  (2021-02-15T16:04:10Z)
• WAFFLe: Weight Anonymized Factorization for Federated Learning [88.44939168851721]
  In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices. We propose Weight Anonymized Factorization for Federated Learning (WAFFLe), an approach that combines the Indian Buffet Process with a shared dictionary of weight factors for neural networks.
  arXiv  Detail & Related papers  (2020-08-13T04:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.