Gradual Federated Learning with Simulated Annealing

• URL: http://arxiv.org/abs/2110.05178v1
• Date: Mon, 11 Oct 2021 11:57:56 GMT
• Title: Gradual Federated Learning with Simulated Annealing
• Authors: Luong Trung Nguyen, Junhan Kim, and Byonghyo Shim
• Abstract summary: Federated averaging (FedAvg) is a popular federated learning (FL) technique that updates the global model by averaging local models. In this paper, we propose a new FL technique based on simulated annealing. We show that SAFL outperforms the conventional FedAvg technique in terms of the convergence speed and the classification accuracy.
• Score: 26.956032164461377
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated averaging (FedAvg) is a popular federated learning (FL) technique that updates the global model by averaging local models and then transmits the updated global model to devices for their local model update. One main limitation of FedAvg is that the average-based global model is not necessarily better than local models in the early stage of the training process so that FedAvg might diverge in realistic scenarios, especially when the data is non-identically distributed across devices and the number of data samples varies significantly from device to device. In this paper, we propose a new FL technique based on simulated annealing. The key idea of the proposed technique, henceforth referred to as \textit{simulated annealing-based FL} (SAFL), is to allow a device to choose its local model when the global model is immature. Specifically, by exploiting the simulated annealing strategy, we make each device choose its local model with high probability in early iterations when the global model is immature. From extensive numerical experiments using various benchmark datasets, we demonstrate that SAFL outperforms the conventional FedAvg technique in terms of the convergence speed and the classification accuracy.

Related papers

• Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updates [71.81037644563217]
  Synchronous federated learning (FL) is a popular paradigm for collaborative edge learning. As some of the devices may have limited computational resources and varying availability, FL latency is highly sensitive to stragglers. We propose straggler-aware layer-wise federated learning (SALF) that leverages the optimization procedure of NNs via backpropagation to update the global model in a layer-wise fashion.
  arXiv  Detail & Related papers  (2024-03-27T09:14:36Z)
• Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Network [4.1813760301635705]
  We propose Federated Partial Model Training (FedPMT), where devices with smaller computational capabilities work on partial models and contribute to the global model. As such, all devices in FedPMT prioritize the most crucial parts of the global model. Empirical results show that FedPMT significantly outperforms the existing benchmark FedDrop.
  arXiv  Detail & Related papers  (2023-11-16T16:30:04Z)
• Fed-FSNet: Mitigating Non-I.I.D. Federated Learning via Fuzzy Synthesizing Network [19.23943687834319]
  Federated learning (FL) has emerged as a promising privacy-preserving distributed machine learning framework. We propose a novel FL training framework, dubbed Fed-FSNet, using a properly designed Fuzzy Synthesizing Network (FSNet) to mitigate the Non-I.I.D. at-the-source issue.
  arXiv  Detail & Related papers  (2022-08-21T18:40:51Z)
• FedEntropy: Efficient Device Grouping for Federated Learning Using Maximum Entropy Judgment [10.507028041279048]
  Federated Learning (FL) has attracted steadily increasing attentions as a promising distributed machine learning paradigm. FL inherently suffers from low classification accuracy in non-IID scenarios. We present an effective FL method named FedEntropy with a novel dynamic device grouping scheme.
  arXiv  Detail & Related papers  (2022-05-24T12:45:17Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Federated and Generalized Person Re-identification through Domain and Feature Hallucinating [88.77196261300699]
  We study the problem of federated domain generalization (FedDG) for person re-identification (re-ID) We propose a novel method, called "Domain and Feature Hallucinating (DFH)", to produce diverse features for learning generalized local and global models. Our method achieves the state-of-the-art performance for FedDG on four large-scale re-ID benchmarks.
  arXiv  Detail & Related papers  (2022-03-05T09:15:13Z)
• FedCAT: Towards Accurate Federated Learning via Device Concatenation [4.416919766772866]
  Federated Learning (FL) enables all the involved devices to train a global model collaboratively without exposing their local data privacy. For non-IID scenarios, the classification accuracy of FL models decreases drastically due to the weight divergence caused by data heterogeneity. We introduce a novel FL approach named Fed-Cat that can achieve high model accuracy based on our proposed device selection strategy and device concatenation-based local training method.
  arXiv  Detail & Related papers  (2022-02-23T10:08:43Z)
• Federated Learning With Quantized Global Model Updates [84.55126371346452]
  We study federated learning, which enables mobile devices to utilize their local datasets to train a global model. We introduce a lossy FL (LFL) algorithm, in which both the global model and the local model updates are quantized before being transmitted.
  arXiv  Detail & Related papers  (2020-06-18T16:55:20Z)
• UVeQFed: Universal Vector Quantization for Federated Learning [179.06583469293386]
  Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their possibly private labeled data. In FL, each user trains its copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion.
  arXiv  Detail & Related papers  (2020-06-05T07:10:22Z)
• Think Locally, Act Globally: Federated Learning with Local and Global Representations [92.68484710504666]
  Federated learning is a method of training models on private data distributed over multiple devices. We propose a new federated learning algorithm that jointly learns compact local representations on each device. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key.
  arXiv  Detail & Related papers  (2020-01-06T12:40:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.