Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?

• URL: http://arxiv.org/abs/2409.03863v1
• Date: Thu, 5 Sep 2024 19:00:18 GMT
• Title: Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?
• Authors: Peizhong Ju, Haibo Yang, Jia Liu, Yingbin Liang, Ness Shroff,
• Abstract summary: Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention.
• Score: 50.03434441234569
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms along with their optimization analyses have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention. This lack of investigation can be attributed to the complex interplay between data heterogeneity and infrequent communication due to the local updates within the FL framework. This motivates us to investigate a fundamental question in FL: Can we quantify the impact of data heterogeneity and local updates on the generalization performance for FL as the learning process evolves? To this end, we conduct a comprehensive theoretical study of FL's generalization performance using a linear model as the first step, where the data heterogeneity is considered for both the stationary and online/non-stationary cases. By providing closed-form expressions of the model error, we rigorously quantify the impact of the number of the local updates (denoted as $K$) under three settings ($K=1$, $K<\infty$, and $K=\infty$) and show how the generalization performance evolves with the number of rounds $t$. Our investigation also provides a comprehensive understanding of how different configurations (including the number of model parameters $p$ and the number of training samples $n$) contribute to the overall generalization performance, thus shedding new insights (such as benign overfitting) for implementing FL over networks.

Related papers

• FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillation [32.305134875959226]
  Federated learning (FL) is a privacy-preserving paradigm that enables distributed clients to collaboratively train models with a central server. We propose FedHPL, a parameter-efficient unified $textbfFed$erated learning framework for $textbfH$eterogeneous settings. We show that our framework outperforms state-of-the-art FL approaches, with less overhead and training rounds.
  arXiv  Detail & Related papers  (2024-05-27T15:25:32Z)
• Multi-level Personalized Federated Learning on Heterogeneous and Long-Tailed Data [10.64629029156029]
  We introduce an innovative personalized Federated Learning framework, Multi-level Personalized Federated Learning (MuPFL) MuPFL integrates three pivotal modules: Biased Activation Value Dropout (BAVD), Adaptive Cluster-based Model Update (ACMU) and Prior Knowledge-assisted Fine-tuning (PKCF) Experiments on diverse real-world datasets show that MuPFL consistently outperforms state-of-the-art baselines, even under extreme non-i.i.d. and long-tail conditions.
  arXiv  Detail & Related papers  (2024-05-10T11:52:53Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning [60.41501515192088]
  Federated Learning (FL) has become a popular distributed learning paradigm that involves multiple clients training a global model collaboratively. The data samples usually follow a long-tailed distribution in the real world, and FL on the decentralized and long-tailed data yields a poorly-behaved global model. In this work, we integrate the local real data with the global gradient prototypes to form the local balanced datasets.
  arXiv  Detail & Related papers  (2023-01-25T03:18:10Z)
• When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods [22.16636947999123]
  We introduce various parameter-efficient tuning (PETuning) methods into federated learning. Specifically, we provide a holistic empirical study of representative PLMs tuning methods in FL. Overall communication overhead can be significantly reduced by locally tuning and globally aggregating lightweight model parameters.
  arXiv  Detail & Related papers  (2022-12-20T06:44:32Z)
• FL Games: A Federated Learning Framework for Distribution Shifts [71.98708418753786]
  Federated learning aims to train predictive models for data that is distributed across clients, under the orchestration of a server. We propose FL GAMES, a game-theoretic framework for federated learning that learns causal features that are invariant across clients.
  arXiv  Detail & Related papers  (2022-10-31T22:59:03Z)
• On the Importance and Applicability of Pre-Training for Federated Learning [28.238484580662785]
  We conduct a systematic study to explore pre-training for federated learning. We find that pre-training can improve FL, but also close its accuracy gap to the counterpart centralized learning. We conclude our paper with an attempt to understand the effect of pre-training on FL.
  arXiv  Detail & Related papers  (2022-06-23T06:02:33Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)
• Federated learning with hierarchical clustering of local updates to improve training on non-IID data [3.3517146652431378]
  We show that learning a single joint model is often not optimal in the presence of certain types of non-iid data. We present a modification to FL by introducing a hierarchical clustering step (FL+HC) We show how FL+HC allows model training to converge in fewer communication rounds compared to FL without clustering.
  arXiv  Detail & Related papers  (2020-04-24T15:16:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.