Take History as a Mirror in Heterogeneous Federated Learning
- URL: http://arxiv.org/abs/2312.10425v1
- Date: Sat, 16 Dec 2023 11:40:49 GMT
- Title: Take History as a Mirror in Heterogeneous Federated Learning
- Authors: Xiaorui Jiang, Hengwei Xu, Yu Gao, Yong Liao, Pengyuan Zhou
- Abstract summary: Federated Learning (FL) allows several clients to cooperatively train machine learning models without disclosing the raw data.
In this work, we propose a novel asynchronous FL framework called Federated Historical Learning (FedHist)
FedHist effectively addresses the challenges posed by both Non-IID data and gradient staleness.
- Score: 9.187993085263209
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated Learning (FL) allows several clients to cooperatively train machine
learning models without disclosing the raw data. In practice, due to the system
and statistical heterogeneity among devices, synchronous FL often encounters
the straggler effect. In contrast, asynchronous FL can mitigate this problem,
making it suitable for scenarios involving numerous participants. However,
Non-IID data and stale models present significant challenges to asynchronous
FL, as they would diminish the practicality of the global model and even lead
to training failures. In this work, we propose a novel asynchronous FL
framework called Federated Historical Learning (FedHist), which effectively
addresses the challenges posed by both Non-IID data and gradient staleness.
FedHist enhances the stability of local gradients by performing weighted fusion
with historical global gradients cached on the server. Relying on hindsight, it
assigns aggregation weights to each participant in a multi-dimensional manner
during each communication round. To further enhance the efficiency and
stability of the training process, we introduce an intelligent $\ell_2$-norm
amplification scheme, which dynamically regulates the learning progress based
on the $\ell_2$-norms of the submitted gradients. Extensive experiments
demonstrate that FedHist outperforms state-of-the-art methods in terms of
convergence performance and test accuracy.
Related papers
- An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets.
Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round.
We introduce FedAF, a novel aggregation-free FL algorithm.
arXiv Detail & Related papers (2024-04-29T05:55:23Z) - Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updates [71.81037644563217]
Synchronous federated learning (FL) is a popular paradigm for collaborative edge learning.
As some of the devices may have limited computational resources and varying availability, FL latency is highly sensitive to stragglers.
We propose straggler-aware layer-wise federated learning (SALF) that leverages the optimization procedure of NNs via backpropagation to update the global model in a layer-wise fashion.
arXiv Detail & Related papers (2024-03-27T09:14:36Z) - FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm.
It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities.
It achieves substantial and consistent improvements over state-of-the-art baselines.
arXiv Detail & Related papers (2024-02-13T20:04:39Z) - Asynchronous Multi-Model Dynamic Federated Learning over Wireless
Networks: Theory, Modeling, and Optimization [20.741776617129208]
Federated learning (FL) has emerged as a key technique for distributed machine learning (ML)
We first formulate rectangular scheduling steps and functions to capture the impact of system parameters on learning performance.
Our analysis sheds light on the joint impact of device training variables and asynchronous scheduling decisions.
arXiv Detail & Related papers (2023-05-22T21:39:38Z) - Depersonalized Federated Learning: Tackling Statistical Heterogeneity by
Alternating Stochastic Gradient Descent [6.394263208820851]
Federated learning (FL) enables devices to train a common machine learning (ML) model for intelligent inference without data sharing.
Raw data held by various cooperativelyicipators are always non-identically distributedly.
We propose a new FL that can significantly statistical optimize by the de-speed of this process.
arXiv Detail & Related papers (2022-10-07T10:30:39Z) - FEDIC: Federated Learning on Non-IID and Long-Tailed Data via Calibrated
Distillation [54.2658887073461]
Dealing with non-IID data is one of the most challenging problems for federated learning.
This paper studies the joint problem of non-IID and long-tailed data in federated learning and proposes a corresponding solution called Federated Ensemble Distillation with Imbalance (FEDIC)
FEDIC uses model ensemble to take advantage of the diversity of models trained on non-IID data.
arXiv Detail & Related papers (2022-04-30T06:17:36Z) - Fine-tuning Global Model via Data-Free Knowledge Distillation for
Non-IID Federated Learning [86.59588262014456]
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint.
We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG)
Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
arXiv Detail & Related papers (2022-03-17T11:18:17Z) - Towards Efficient and Stable K-Asynchronous Federated Learning with
Unbounded Stale Gradients on Non-IID Data [10.299577499118548]
Federated learning (FL) is an emerging privacy-preserving paradigm that enables multiple participants to train a global model without uploading raw data.
This paper proposes a two-stage weighted $K$ asynchronous FL with adaptive learning rate (WKAFL)
Experiments implemented on both benchmark and synthetic FL datasets show that WKAFL has better overall performance compared to existing algorithms.
arXiv Detail & Related papers (2022-03-02T16:17:23Z) - Towards Federated Learning on Time-Evolving Heterogeneous Data [13.080665001587281]
Federated Learning (FL) is an emerging learning paradigm that preserves privacy by ensuring client data locality on edge devices.
Despite recent research efforts on improving the optimization of heterogeneous data, the impact of time-evolving heterogeneous data in real-world scenarios has not been well studied.
We propose Continual Federated Learning (CFL), a flexible framework, to capture the time-evolving heterogeneity of FL.
arXiv Detail & Related papers (2021-12-25T14:58:52Z) - Dynamic Attention-based Communication-Efficient Federated Learning [85.18941440826309]
Federated learning (FL) offers a solution to train a global machine learning model.
FL suffers performance degradation when client data distribution is non-IID.
We propose a new adaptive training algorithm $textttAdaFL$ to combat this degradation.
arXiv Detail & Related papers (2021-08-12T14:18:05Z) - FedSemi: An Adaptive Federated Semi-Supervised Learning Framework [23.90642104477983]
Federated learning (FL) has emerged as an effective technique to co-training machine learning models without actually sharing data and leaking privacy.
Most existing FL methods focus on the supervised setting and ignore the utilization of unlabeled data.
We propose FedSemi, a novel, adaptive, and general framework, which firstly introduces the consistency regularization into FL using a teacher-student model.
arXiv Detail & Related papers (2020-12-06T15:46:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.