Knowledge Rumination for Client Utility Evaluation in Heterogeneous Federated Learning

• URL: http://arxiv.org/abs/2312.10425v2
• Date: Sun, 23 Mar 2025 06:14:35 GMT
• Title: Knowledge Rumination for Client Utility Evaluation in Heterogeneous Federated Learning
• Authors: Xiaorui Jiang, Yu Gao, Hengwei Xu, Qi Zhang, Yong Liao, Pengyuan Zhou,
• Abstract summary: Federated Learning (FL) allows several clients to cooperatively train machine learning models without disclosing the raw data.<n>Non-IID data and stale models pose significant challenges to AFL, as they can diminish the practicality of the global model and even lead to training failures.<n>We propose a novel AFL framework called Federated Historical Learning (FedHist), which effectively addresses the challenges posed by both Non-IID data and gradient staleness.
• Score: 12.50871784200551
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) allows several clients to cooperatively train machine learning models without disclosing the raw data. In practical applications, asynchronous FL (AFL) can address the straggler effect compared to synchronous FL. However, Non-IID data and stale models pose significant challenges to AFL, as they can diminish the practicality of the global model and even lead to training failures. In this work, we propose a novel AFL framework called Federated Historical Learning (FedHist), which effectively addresses the challenges posed by both Non-IID data and gradient staleness based on the concept of knowledge rumination. FedHist enhances the stability of local gradients by performing weighted fusion with historical global gradients cached on the server. Relying on hindsight, it assigns aggregation weights to each participant in a multi-dimensional manner during each communication round. To further enhance the efficiency and stability of the training process, we introduce an intelligent $\ell_2$-norm amplification scheme, which dynamically regulates the learning progress based on the $\ell_2$-norms of the submitted gradients. Extensive experiments indicate FedHist outperforms state-of-the-art methods in terms of convergence performance and test accuracy.

Related papers

• Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data [17.624808621195978]
  We propose a Selective Self-Distillation method for Federated learning (FedSSD) FedSSD imposes adaptive constraints on the local updates by self-distilling the global model's knowledge. It achieves better generalization and robustness in fewer communication rounds, compared with other state-of-the-art FL methods.
  arXiv  Detail & Related papers  (2025-04-20T18:06:55Z)
• Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible [2.663489028501814]
  This paper proposes an asynchronous federated learning version correction algorithm based on knowledge distillation, named FedADT. FedADT applies knowledge distillation before aggregating gradients, using the latest global model to correct outdated information, thus effectively reducing the negative impact of outdated gradients on the training process. We conducted experimental comparisons with several classical algorithms, and the results demonstrate that FedADT achieves significant improvements over other asynchronous methods and outperforms all methods in terms of convergence speed.
  arXiv  Detail & Related papers  (2025-04-05T06:54:13Z)
• SEAFL: Enhancing Efficiency in Semi-Asynchronous Federated Learning through Adaptive Aggregation and Selective Training [26.478852701376294]
  We present em SEAFL, a novel FL framework designed to mitigate both the straggler and the stale model challenges in semi-asynchronous FL. em SEAFL dynamically assigns weights to uploaded models during aggregation based on their staleness and importance to the current global model. We evaluate the effectiveness of em SEAFL through extensive experiments on three benchmark datasets.
  arXiv  Detail & Related papers  (2025-02-22T05:13:53Z)
• Analytic Personalized Federated Meta-Learning [15.1961498951975]
  Analytic Federated Learning (AFL) is an enhanced gradient-free learning (FL) paradigm designed to accelerate training by updating the global model in a single step with closed-form least-square (LS) solutions. We propose FedACnnwise framework in which a layerwise training method is designed by modeling each layer as an LS problem. It generates a personalized model for each client by analytically solving a local objective which bridges the gap between the global model and the individual data degradation.
  arXiv  Detail & Related papers  (2025-02-10T11:27:54Z)
• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models. Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• One-shot Federated Learning via Synthetic Distiller-Distillate Communication [63.89557765137003]
  One-shot Federated learning (FL) is a powerful technology facilitating collaborative training of machine learning models in a single round of communication. We propose FedSD2C, a novel and practical one-shot FL framework designed to address these challenges.
  arXiv  Detail & Related papers  (2024-12-06T17:05:34Z)
• Using Diffusion Models as Generative Replay in Continual Federated Learning -- What will Happen? [28.465434174993007]
  This paper introduces DCFL, a novel framework tailored to address the challenges of continuous learning in distributed learning environments. Our approach harnesses the power of the conditional diffusion model to generate synthetic historical data at each local device during communication.
  arXiv  Detail & Related papers  (2024-11-10T22:48:21Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updates [71.81037644563217]
  Synchronous federated learning (FL) is a popular paradigm for collaborative edge learning. As some of the devices may have limited computational resources and varying availability, FL latency is highly sensitive to stragglers. We propose straggler-aware layer-wise federated learning (SALF) that leverages the optimization procedure of NNs via backpropagation to update the global model in a layer-wise fashion.
  arXiv  Detail & Related papers  (2024-03-27T09:14:36Z)
• FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
  FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm. It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities. It achieves substantial and consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:04:39Z)
• Combating Exacerbated Heterogeneity for Robust Models in Federated Learning [91.88122934924435]
  Combination of adversarial training and federated learning can lead to the undesired robustness deterioration. We propose a novel framework called Slack Federated Adversarial Training (SFAT) We verify the rationality and effectiveness of SFAT on various benchmarked and real-world datasets.
  arXiv  Detail & Related papers  (2023-03-01T06:16:15Z)
• Depersonalized Federated Learning: Tackling Statistical Heterogeneity by Alternating Stochastic Gradient Descent [6.394263208820851]
  Federated learning (FL) enables devices to train a common machine learning (ML) model for intelligent inference without data sharing. Raw data held by various cooperativelyicipators are always non-identically distributedly. We propose a new FL that can significantly statistical optimize by the de-speed of this process.
  arXiv  Detail & Related papers  (2022-10-07T10:30:39Z)
• FEDIC: Federated Learning on Non-IID and Long-Tailed Data via Calibrated Distillation [54.2658887073461]
  Dealing with non-IID data is one of the most challenging problems for federated learning. This paper studies the joint problem of non-IID and long-tailed data in federated learning and proposes a corresponding solution called Federated Ensemble Distillation with Imbalance (FEDIC) FEDIC uses model ensemble to take advantage of the diversity of models trained on non-IID data.
  arXiv  Detail & Related papers  (2022-04-30T06:17:36Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data [10.299577499118548]
  Federated learning (FL) is an emerging privacy-preserving paradigm that enables multiple participants to train a global model without uploading raw data. This paper proposes a two-stage weighted $K$ asynchronous FL with adaptive learning rate (WKAFL) Experiments implemented on both benchmark and synthetic FL datasets show that WKAFL has better overall performance compared to existing algorithms.
  arXiv  Detail & Related papers  (2022-03-02T16:17:23Z)
• Towards Federated Learning on Time-Evolving Heterogeneous Data [13.080665001587281]
  Federated Learning (FL) is an emerging learning paradigm that preserves privacy by ensuring client data locality on edge devices. Despite recent research efforts on improving the optimization of heterogeneous data, the impact of time-evolving heterogeneous data in real-world scenarios has not been well studied. We propose Continual Federated Learning (CFL), a flexible framework, to capture the time-evolving heterogeneity of FL.
  arXiv  Detail & Related papers  (2021-12-25T14:58:52Z)
• Dynamic Attention-based Communication-Efficient Federated Learning [85.18941440826309]
  Federated learning (FL) offers a solution to train a global machine learning model. FL suffers performance degradation when client data distribution is non-IID. We propose a new adaptive training algorithm $textttAdaFL$ to combat this degradation.
  arXiv  Detail & Related papers  (2021-08-12T14:18:05Z)
• FedSemi: An Adaptive Federated Semi-Supervised Learning Framework [23.90642104477983]
  Federated learning (FL) has emerged as an effective technique to co-training machine learning models without actually sharing data and leaking privacy. Most existing FL methods focus on the supervised setting and ignore the utilization of unlabeled data. We propose FedSemi, a novel, adaptive, and general framework, which firstly introduces the consistency regularization into FL using a teacher-student model.
  arXiv  Detail & Related papers  (2020-12-06T15:46:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.