Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization

• URL: http://arxiv.org/abs/2604.12768v1
• Date: Tue, 14 Apr 2026 14:08:57 GMT
• Title: Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization
• Abstract summary: Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model.<n>Previous works have implicitly studied that FL suffers from the client-drift'' problem, which is caused by the inconsistent optimum across local clients.<n>To alleviate the negative impact of client drift'' and explore its substance in FL, we first propose an efficient FL algorithm FedInit.
• Score: 60.39868436334864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset. Previous works have implicitly studied that FL suffers from the ``client-drift'' problem, which is caused by the inconsistent optimum across local clients. However, till now it still lacks solid theoretical analysis to explain the impact of this local inconsistency. To alleviate the negative impact of ``client drift'' and explore its substance in FL, in this paper, we first propose an efficient FL algorithm FedInit, which allows employing the personalized relaxed initialization state at the beginning of each local training stage. Specifically, FedInit initializes the local state by moving away from the current global state towards the reverse direction of the latest local state. Moreover, to further understand how inconsistency disrupts performance in FL, we introduce the excess risk analysis and study the divergence term to investigate the test error in FL. Our studies show that optimization error is not sensitive to this local inconsistency, while it mainly affects the generalization error bound. Extensive experiments are conducted to validate its efficiency. The proposed FedInit method could achieve comparable results compared to several advanced benchmarks without any additional training or communication costs. Meanwhile, the stage-wise personalized relaxed initialization could also be incorporated into several current advanced algorithms to achieve higher generalization performance in the FL paradigm.

Related papers

• FedNSAM:Consistency of Local and Global Flatness for Federated Learning [26.41380732455181]
  We propose a novel textbfFedNSAM algorithm that accelerates the SAM algorithm by introducing global Nesterov momentum into the local update.<n>textbfFedNSAM uses the global Nesterov momentum as the direction of local estimation of client global perturbations and extrapolation.<n> Empirically, we conduct comprehensive experiments on CNN and Transformer models to verify the superior performance and efficiency of textbfFedNSAM.
  arXiv  Detail & Related papers  (2026-02-27T09:07:47Z)
• Federated Learning with Feedback Alignment [1.879054346828946]
  Federated Learning (FL) enables collaborative training across multiple clients while preserving data privacy.<n>We introduce Federated Learning with Feedback Alignment (FLFA), a novel framework that integrates feedback alignment into FL.<n>FLFA uses the global model's weights as a shared feedback matrix during local training's backward pass, aligning local updates with the global model efficiently.
  arXiv  Detail & Related papers  (2025-12-14T16:59:55Z)
• Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning? [50.03434441234569]
  Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention.
  arXiv  Detail & Related papers  (2024-09-05T19:00:18Z)
• Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization [81.32266996009575]
  In federated learning (FL), the multi-step update and data heterogeneity among clients often lead to a loss landscape with sharper minima. We propose FedLESAM, a novel algorithm that locally estimates the direction of global perturbation on client side.
  arXiv  Detail & Related papers  (2024-05-29T08:46:21Z)
• Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization [84.42306265220274]
  Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model. Previous works have implicitly studied that FL suffers from the client-drift'' problem, which is caused by the inconsistent optimum across local clients. To alleviate the negative impact of the client drift'' and explore its substance in FL, we first design an efficient FL algorithm textitFedInit.
  arXiv  Detail & Related papers  (2023-06-09T06:55:15Z)
• FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy [84.45004766136663]
  Federated learning is an emerging distributed machine learning framework. It suffers from the non-vanishing biases introduced by the local inconsistent optimal and the rugged client-drifts by the local over-fitting. We propose a novel and practical method, FedSpeed, to alleviate the negative impacts posed by these problems.
  arXiv  Detail & Related papers  (2023-02-21T03:55:29Z)
• On the Importance and Applicability of Pre-Training for Federated Learning [28.238484580662785]
  We conduct a systematic study to explore pre-training for federated learning. We find that pre-training can improve FL, but also close its accuracy gap to the counterpart centralized learning. We conclude our paper with an attempt to understand the effect of pre-training on FL.
  arXiv  Detail & Related papers  (2022-06-23T06:02:33Z)
• Generalized Federated Learning via Sharpness Aware Minimization [22.294290071999736]
  We propose a general, effective algorithm, textttFedSAM, based on Sharpness Aware Minimization (SAM) local, and develop a momentum FL algorithm to bridge local and global models. Empirically, our proposed algorithms substantially outperform existing FL studies and significantly decrease the learning deviation.
  arXiv  Detail & Related papers  (2022-06-06T13:54:41Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)
• Critical Learning Periods in Federated Learning [11.138980572551066]
  Federated learning (FL) is a popular technique to train machine learning (ML) models with decentralized data. We show that the final test accuracy of FL is dramatically affected by the early phase of the training process.
  arXiv  Detail & Related papers  (2021-09-12T21:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.