Overcoming the Challenges of Batch Normalization in Federated Learning

• URL: http://arxiv.org/abs/2405.14670v1
• Date: Thu, 23 May 2024 15:07:21 GMT
• Title: Overcoming the Challenges of Batch Normalization in Federated Learning
• Authors: Rachid Guerraoui, Rafael Pinot, Geovani Rizk, John Stephan, François Taiani,
• Abstract summary: We introduce Federated BatchNorm (FBN), a novel scheme that restores the benefits of batch normalization in federated learning. Essentially, FBN ensures that the batch normalization during training is consistent with what would be achieved in a centralized execution. We show that, with a slight increase in complexity, we can robustify FBN to mitigate erroneous statistics and potentially adversarial attacks.
• Score: 9.94980188821453
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Batch normalization has proven to be a very beneficial mechanism to accelerate the training and improve the accuracy of deep neural networks in centralized environments. Yet, the scheme faces significant challenges in federated learning, especially under high data heterogeneity. Essentially, the main challenges arise from external covariate shifts and inconsistent statistics across clients. We introduce in this paper Federated BatchNorm (FBN), a novel scheme that restores the benefits of batch normalization in federated learning. Essentially, FBN ensures that the batch normalization during training is consistent with what would be achieved in a centralized execution, hence preserving the distribution of the data, and providing running statistics that accurately approximate the global statistics. FBN thereby reduces the external covariate shift and matches the evaluation performance of the centralized setting. We also show that, with a slight increase in complexity, we can robustify FBN to mitigate erroneous statistics and potentially adversarial attacks.

Related papers

• Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning [34.07540860626276]
  Batch Normalisation (BN) is widely used in deep neural network training to harmonise the input-output distributions for each batch of data.<n>In this work, we develop a customised normalisation approach, Hybrid Batch Normalisation (HBN)<n>HBN separates the update of statistical parameters from that of learnable parameters, obtaining unbiased estimates of global statistical parameters in distributed scenarios.
  arXiv  Detail & Related papers  (2025-05-28T01:46:34Z)
• Feature Norm Regularized Federated Learning: Transforming Skewed Distributions into Global Insights [16.039822050613022]
  This work introduces the Feature Norm Regularized Federated Learning (FNR-FL) algorithm. FNR-FL incorporates class average feature norms to enhance model accuracy and convergence in non-i.i.d. scenarios. We show that FNR-FL exhibits a substantial 66.24% improvement in accuracy and a significant 11.40% reduction in training time.
  arXiv  Detail & Related papers  (2023-12-12T03:09:37Z)
• Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation [67.77048565738728]
  Continual learning involves learning a sequence of tasks and balancing their knowledge appropriately. We propose Adaptive Balance of BN (AdaB$2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions. Our approach achieves significant performance gains across a wide range of benchmarks.
  arXiv  Detail & Related papers  (2023-10-13T04:50:40Z)
• FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation [95.85026305874824]
  We introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency.
  arXiv  Detail & Related papers  (2022-12-14T13:57:01Z)
• FedGen: Generalizable Federated Learning for Sequential Data [8.784435748969806]
  In many real-world distributed settings, spurious correlations exist due to biases and data sampling issues. We present a generalizable federated learning framework called FedGen, which allows clients to identify and distinguish between spurious and invariant features. We show that FedGen results in models that achieve significantly better generalization and can outperform the accuracy of current federated learning approaches by over 24%.
  arXiv  Detail & Related papers  (2022-11-03T15:48:14Z)
• Debiasing Neural Retrieval via In-batch Balancing Regularization [25.941718123899356]
  We develop a differentiable textitnormed Pairwise Ranking Fairness (nPRF) and leverage the T-statistics on top of nPRF to improve fairness. Our method with nPRF achieves significantly less bias with minimal degradation in ranking performance compared with the baseline.
  arXiv  Detail & Related papers  (2022-05-18T22:57:15Z)
• Heterogeneous Federated Learning via Grouped Sequential-to-Parallel Training [60.892342868936865]
  Federated learning (FL) is a rapidly growing privacy-preserving collaborative machine learning paradigm. We propose a data heterogeneous-robust FL approach, FedGSP, to address this challenge. We show that FedGSP improves the accuracy by 3.7% on average compared with seven state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-01-31T03:15:28Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)
• Decentralized Local Stochastic Extra-Gradient for Variational Inequalities [125.62877849447729]
  We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices. We make a very general assumption on the computational network that covers the settings of fully decentralized calculations. We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
  arXiv  Detail & Related papers  (2021-06-15T17:45:51Z)
• Double Forward Propagation for Memorized Batch Normalization [68.34268180871416]
  Batch Normalization (BN) has been a standard component in designing deep neural networks (DNNs) We propose a memorized batch normalization (MBN) which considers multiple recent batches to obtain more accurate and robust statistics. Compared to related methods, the proposed MBN exhibits consistent behaviors in both training and inference.
  arXiv  Detail & Related papers  (2020-10-10T08:48:41Z)
• Robustness and Personalization in Federated Learning: A Unified Approach via Regularization [4.7234844467506605]
  We present a class of methods for robust, personalized federated learning, called Fed+. The principal advantage of Fed+ is to better accommodate the real-world characteristics found in federated training. We demonstrate the benefits of Fed+ through extensive experiments on benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-14T10:04:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.