Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learning

• URL: http://arxiv.org/abs/2310.01366v1
• Date: Mon, 2 Oct 2023 17:30:14 GMT
• Title: Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learning
• Authors: Debora Caldarola, Barbara Caputo, Marco Ciccone
• Abstract summary: Federated Learning (FL) aims to learn a global model from distributed users while protecting their privacy. We propose WIMA (Window-based Model Averaging), which aggregates global models from different rounds using a window-based approach. Our experiments demonstrate the robustness of WIMA against distribution shifts and bad client sampling, resulting in smoother and more stable learning trends.
• Score: 29.140054600391917
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) aims to learn a global model from distributed users while protecting their privacy. However, when data are distributed heterogeneously the learning process becomes noisy, unstable, and biased towards the last seen clients' data, slowing down convergence. To address these issues and improve the robustness and generalization capabilities of the global model, we propose WIMA (Window-based Model Averaging). WIMA aggregates global models from different rounds using a window-based approach, effectively capturing knowledge from multiple users and reducing the bias from the last ones. By adopting a windowed view on the rounds, WIMA can be applied from the initial stages of training. Importantly, our method introduces no additional communication or client-side computation overhead. Our experiments demonstrate the robustness of WIMA against distribution shifts and bad client sampling, resulting in smoother and more stable learning trends. Additionally, WIMA can be easily integrated with state-of-the-art algorithms. We extensively evaluate our approach on standard FL benchmarks, demonstrating its effectiveness.

Related papers

• FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization [11.040916982022978]
  Federated Learning (FL) enables collaborative training of machine learning models on decentralized data. Data across clients often differs significantly due to class imbalance, feature distribution skew, sample size imbalance, and other phenomena. We propose a novel Bayesian PFL framework using bi-level optimization to tackle the data heterogeneity challenges.
  arXiv  Detail & Related papers  (2024-05-29T11:28:06Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Client-side Gradient Inversion Against Federated Learning from Poisoning [59.74484221875662]
  Federated Learning (FL) enables distributed participants to train a global model without sharing data directly to a central server. Recent studies have revealed that FL is vulnerable to gradient inversion attack (GIA), which aims to reconstruct the original training samples. We propose Client-side poisoning Gradient Inversion (CGI), which is a novel attack method that can be launched from clients.
  arXiv  Detail & Related papers  (2023-09-14T03:48:27Z)
• Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
  Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. We find that the difference in logits between the local and global models increases as the model is continuously updated. We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
  arXiv  Detail & Related papers  (2023-08-20T04:41:01Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning [9.975023463908496]
  Federated Learning (FL) is a machine learning paradigm that enables clients to jointly train a global model by aggregating the locally trained models without sharing any local training data. We propose a novel regularization technique based on adaptive self-distillation (ASD) for training models on the client side. Our regularization scheme adaptively adjusts to the client's training data based on the global model entropy and the client's label distribution.
  arXiv  Detail & Related papers  (2023-05-31T07:00:42Z)
• Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms [29.636944156801327]
  Multiple clients collaboratively train one global model without sharing their semantic parsing data. Lorar adjusts each client's contribution to the global model update based on its training loss reduction during each round. Clients with smaller datasets enjoy larger performance gains.
  arXiv  Detail & Related papers  (2023-05-26T19:25:49Z)
• Confidence-aware Personalized Federated Learning via Variational Expectation Maximization [34.354154518009956]
  We present a novel framework for personalized Federated Learning (PFL) PFL is a distributed learning scheme to train a shared model across clients. We present a novel framework for PFL based on hierarchical modeling and variational inference.
  arXiv  Detail & Related papers  (2023-05-21T20:12:27Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Gradient Masked Averaging for Federated Learning [24.687254139644736]
  Federated learning allows a large number of clients with heterogeneous data to coordinate learning of a unified global model. Standard FL algorithms involve averaging of model parameters or gradient updates to approximate the global model at the server. We propose a gradient masked averaging approach for FL as an alternative to the standard averaging of client updates.
  arXiv  Detail & Related papers  (2022-01-28T08:42:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.