Federated Learning Hyper-Parameter Tuning from a System Perspective

• URL: http://arxiv.org/abs/2211.13656v1
• Date: Thu, 24 Nov 2022 15:15:28 GMT
• Title: Federated Learning Hyper-Parameter Tuning from a System Perspective
• Authors: Huanle Zhang and Lei Fu and Mi Zhang and Pengfei Hu and Xiuzhen Cheng and Prasant Mohapatra and Xin Liu
• Abstract summary: Federated learning (FL) is a distributed model training paradigm that preserves clients' data privacy. Current practice of manually selecting FL hyper- parameters imposes a heavy burden on FL practitioners. We propose FedTune, an automatic FL hyper- parameter tuning algorithm tailored to applications' diverse system requirements.
• Score: 23.516484538620745
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) is a distributed model training paradigm that preserves clients' data privacy. It has gained tremendous attention from both academia and industry. FL hyper-parameters (e.g., the number of selected clients and the number of training passes) significantly affect the training overhead in terms of computation time, transmission time, computation load, and transmission load. However, the current practice of manually selecting FL hyper-parameters imposes a heavy burden on FL practitioners because applications have different training preferences. In this paper, we propose FedTune, an automatic FL hyper-parameter tuning algorithm tailored to applications' diverse system requirements in FL training. FedTune iteratively adjusts FL hyper-parameters during FL training and can be easily integrated into existing FL systems. Through extensive evaluations of FedTune for diverse applications and FL aggregation algorithms, we show that FedTune is lightweight and effective, achieving 8.48%-26.75% system overhead reduction compared to using fixed FL hyper-parameters. This paper assists FL practitioners in designing high-performance FL training solutions. The source code of FedTune is available at https://github.com/DataSysTech/FedTune.

Related papers

• FedHQ: Hybrid Runtime Quantization for Federated Learning [18.039278211314205]
  Federated Learning (FL) is a decentralized model training approach that preserves data privacy but struggles with low efficiency.<n>This paper proposes a hybrid quantitation approach combining PTQ and QAT for FL systems.<n>Experiments show that FedHQ achieves up to 2.47x times training acceleration and up to 11.15% accuracy improvement and negligible extra overhead.
  arXiv  Detail & Related papers  (2025-05-17T12:30:27Z)
• A Survey on Efficient Federated Learning Methods for Foundation Model Training [62.473245910234304]
  Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
  arXiv  Detail & Related papers  (2024-01-09T10:22:23Z)
• MetisFL: An Embarrassingly Parallelized Controller for Scalable & Efficient Federated Learning Workflows [1.9874264019909988]
  A Federated Learning (FL) system typically consists of two core processing entities: the federation controller and the learners. To meet this need, we designed and developed a novel FL system called MetisFL, where the federation controller is the first-class citizen. MetisFL re-engineers all the operations conducted by the federation controller to accelerate the training of large-scale FL.
  arXiv  Detail & Related papers  (2023-11-01T07:01:19Z)
• FLGo: A Fully Customizable Federated Learning Platform [23.09038374160798]
  We propose a novel lightweight Federated learning platform called FLGo. Our platform offers 40+ benchmarks, 20+ algorithms, and 2 system simulators as out-of-the-box plugins. We also develop a range of experimental tools, including parallel acceleration, experiment tracker and parameters auto-tuning.
  arXiv  Detail & Related papers  (2023-06-21T07:55:29Z)
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence [83.58839320635956]
  Federated Learning (FL) can be used in mobile edge networks to train machine learning models in a distributed manner. Recent FL has been interpreted within a Model-Agnostic Meta-Learning (MAML) framework, which brings FL significant advantages in fast adaptation and convergence over heterogeneous datasets. This paper addresses how much benefit MAML brings to FL and how to maximize such benefit over mobile edge networks.
  arXiv  Detail & Related papers  (2023-03-23T02:42:10Z)
• Performance Optimization for Variable Bitwidth Federated Learning in Wireless Networks [103.22651843174471]
  This paper considers improving wireless communication and computation efficiency in federated learning (FL) via model quantization. In the proposed bitwidth FL scheme, edge devices train and transmit quantized versions of their local FL model parameters to a coordinating server, which aggregates them into a quantized global model and synchronizes the devices. We show that the FL training process can be described as a Markov decision process and propose a model-based reinforcement learning (RL) method to optimize action selection over iterations.
  arXiv  Detail & Related papers  (2022-09-21T08:52:51Z)
• Fast Server Learning Rate Tuning for Coded Federated Dropout [3.9653673778225946]
  Federated Dropout (FD) is a technique that improves the communication efficiency of a FL session. We leverage coding theory to enhance FD by allowing a different sub-model to be used at each client. For the EMNIST dataset, our mechanism achieves 99.6 % of the final accuracy of the no dropout case.
  arXiv  Detail & Related papers  (2022-01-26T16:19:04Z)
• Automatic Tuning of Federated Learning Hyper-Parameters from System Perspective [15.108050457914516]
  Federated learning (FL) is a distributed model training paradigm that preserves clients' data privacy. We propose FedTuning, an automatic FL hyper- parameter tuning algorithm tailored to applications' diverse system requirements of FL training. FedTuning is lightweight and flexible, achieving an average of 41% improvement for different training preferences on time, computation, and communication.
  arXiv  Detail & Related papers  (2021-10-06T20:43:25Z)
• Critical Learning Periods in Federated Learning [11.138980572551066]
  Federated learning (FL) is a popular technique to train machine learning (ML) models with decentralized data. We show that the final test accuracy of FL is dramatically affected by the early phase of the training process.
  arXiv  Detail & Related papers  (2021-09-12T21:06:07Z)
• Over-the-Air Federated Learning from Heterogeneous Data [107.05618009955094]
  Federated learning (FL) is a framework for distributed learning of centralized models. We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local gradient descent (SGD) FL algorithm. We numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL.
  arXiv  Detail & Related papers  (2020-09-27T08:28:25Z)
• Delay Minimization for Federated Learning Over Wireless Communication Networks [172.42768672943365]
  The problem of delay computation for federated learning (FL) over wireless communication networks is investigated. A bisection search algorithm is proposed to obtain the optimal solution. Simulation results show that the proposed algorithm can reduce delay by up to 27.3% compared to conventional FL methods.
  arXiv  Detail & Related papers  (2020-07-05T19:00:07Z)
• Convergence Time Optimization for Federated Learning over Wireless Networks [160.82696473996566]
  A wireless network is considered in which wireless users transmit their local FL models (trained using their locally collected data) to a base station (BS) The BS, acting as a central controller, generates a global FL model using the received local FL models and broadcasts it back to all users. Due to the limited number of resource blocks (RBs) in a wireless network, only a subset of users can be selected to transmit their local FL model parameters to the BS. Since each user has unique training data samples, the BS prefers to include all local user FL models to generate a converged global FL model.
  arXiv  Detail & Related papers  (2020-01-22T01:55:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.