FedPop: Federated Population-based Hyperparameter Tuning

• URL: http://arxiv.org/abs/2308.08634v3
• Date: Tue, 23 Jul 2024 09:42:29 GMT
• Title: FedPop: Federated Population-based Hyperparameter Tuning
• Authors: Haokun Chen, Denis Krompass, Jindong Gu, Volker Tresp,
• Abstract summary: Federated Learning (FL) is a distributed machine learning (ML) paradigm, in which multiple clients collaboratively train ML models without centralizing their local data. Despite extensive research on tuning HPs for centralized ML, these methods yield suboptimal results when employed in FL. This is mainly because their "training-after-tuning" framework is unsuitable for FL with limited client power.
• Score: 30.45354486897489
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) is a distributed machine learning (ML) paradigm, in which multiple clients collaboratively train ML models without centralizing their local data. Similar to conventional ML pipelines, the client local optimization and server aggregation procedure in FL are sensitive to the hyperparameter (HP) selection. Despite extensive research on tuning HPs for centralized ML, these methods yield suboptimal results when employed in FL. This is mainly because their "training-after-tuning" framework is unsuitable for FL with limited client computation power. While some approaches have been proposed for HP-Tuning in FL, they are limited to the HPs for client local updates. In this work, we propose a novel HP-tuning algorithm, called Federated Population-based Hyperparameter Tuning (FedPop), to address this vital yet challenging problem. FedPop employs population-based evolutionary algorithms to optimize the HPs, which accommodates various HP types at both the client and server sides. Compared with prior tuning methods, FedPop employs an online "tuning-while-training" framework, offering computational efficiency and enabling the exploration of a broader HP search space. Our empirical validation on the common FL benchmarks and complex real-world FL datasets, including full-sized Non-IID ImageNet-1K, demonstrates the effectiveness of the proposed method, which substantially outperforms the concurrent state-of-the-art HP-tuning methods in FL.

Related papers

• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. Experiments show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• How to Privately Tune Hyperparameters in Federated Learning? Insights from a Benchmark Study [1.4968312514344115]
  We use PrivTuna to implement privacy-preserving federated averaging and density-based clustering. PrivTuna is a novel framework for privacy-preserving HP tuning using multiparty homomorphic encryption.
  arXiv  Detail & Related papers  (2024-02-25T13:25:51Z)
• Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [71.87335804334616]
  Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. The training process of Large Language Models (LLMs) generally incurs the update of significant parameters. This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
  arXiv  Detail & Related papers  (2023-10-23T16:37:59Z)
• FedAVO: Improving Communication Efficiency in Federated Learning with African Vultures Optimizer [0.0]
  Federated Learning (FL) is a distributed machine learning technique. In this paper, we introduce FedAVO, a novel FL algorithm that enhances communication effectiveness. We show that FedAVO achieves significant improvement in terms of model accuracy and communication round.
  arXiv  Detail & Related papers  (2023-05-02T02:04:19Z)
• Federated Learning with Flexible Control [30.65854375019346]
  Federated learning (FL) enables distributed model training from local data collected by users. In distributed systems with constrained resources and potentially high dynamics, e.g., mobile edge networks, the efficiency of FL is an important problem. We propose FlexFL - an FL algorithm with multiple options that can be adjusted flexibly.
  arXiv  Detail & Related papers  (2022-12-16T14:21:29Z)
• Performance Optimization for Variable Bitwidth Federated Learning in Wireless Networks [103.22651843174471]
  This paper considers improving wireless communication and computation efficiency in federated learning (FL) via model quantization. In the proposed bitwidth FL scheme, edge devices train and transmit quantized versions of their local FL model parameters to a coordinating server, which aggregates them into a quantized global model and synchronizes the devices. We show that the FL training process can be described as a Markov decision process and propose a model-based reinforcement learning (RL) method to optimize action selection over iterations.
  arXiv  Detail & Related papers  (2022-09-21T08:52:51Z)
• FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization [50.12374973760274]
  We propose and implement a benchmark suite FedHPO-B that incorporates comprehensive FL tasks, enables efficient function evaluations, and eases continuing extensions. We also conduct extensive experiments based on FedHPO-B to benchmark a few HPO methods.
  arXiv  Detail & Related papers  (2022-06-08T15:29:10Z)
• Single-shot Hyper-parameter Optimization for Federated Learning: A General Algorithm & Analysis [20.98323380319439]
  We introduce Federated Loss SuRface Aggregation (FLoRA), a general FL-HPO solution framework. FLoRA enables single-shot FL-HPO: identifying a single set of good hyper- parameters that are subsequently used in a single FL training. Our empirical evaluation of FLoRA for multiple ML algorithms on seven OpenML datasets demonstrates significant model accuracy improvements over the considered baseline.
  arXiv  Detail & Related papers  (2022-02-16T21:14:34Z)
• FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning [19.854596038293277]
  We introduce Federated Loss suRface Aggregation (FLoRA), the first FL-HPO solution framework. The framework enables single-shot FL-HPO solutions with minimal additional communication overhead. Our empirical evaluation of FLoRA for Gradient Boosted Decision Trees on seven OpenML data sets demonstrates significant model accuracy improvements.
  arXiv  Detail & Related papers  (2021-12-15T23:18:32Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.