Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism

• URL: http://arxiv.org/abs/2411.12244v1
• Date: Tue, 19 Nov 2024 05:49:00 GMT
• Title: Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism
• Authors: Yasaman Saadati, M. Hadi Amini,
• Abstract summary: Federated Learning (FL) is a decentralized learning approach that protects sensitive information by utilizing local model parameters rather than sharing clients' raw datasets. This paper investigates the deployment and integration of two lightweight Hyper- Optimization (HPO) tools, Raytune and Optuna, within the context of FL settings. To this end, both local and global feedback mechanisms are integrated to limit the search space and expedite the HPO process.
• Score: 0.48342038441006796
• License:
• Abstract: Federated Learning (FL) is a decentralized learning approach that protects sensitive information by utilizing local model parameters rather than sharing clients' raw datasets. While this privacy-preserving method is widely employed across various applications, it still requires significant development and optimization. Automated Machine Learning (Auto-ML) has been adapted for reducing the need for manual adjustments. Previous studies have explored the integration of AutoML with different FL algorithms to evaluate their effectiveness in enhancing FL settings. However, Automated FL (Auto-FL) faces additional challenges due to the involvement of a large cohort of clients and global training rounds between clients and the server, rendering the tuning process time-consuming and nearly impossible on resource-constrained edge devices (e.g., IoT devices). This paper investigates the deployment and integration of two lightweight Hyper-Parameter Optimization (HPO) tools, Raytune and Optuna, within the context of FL settings. A step-wise feedback mechanism has also been designed to accelerate the hyper-parameter tuning process and coordinate AutoML toolkits with the FL server. To this end, both local and global feedback mechanisms are integrated to limit the search space and expedite the HPO process. Further, a novel client selection technique is introduced to mitigate the straggler effect in Auto-FL. The selected hyper-parameter tuning tools are evaluated using two benchmark datasets, FEMNIST, and CIFAR10. Further, the paper discusses the essential properties of successful HPO tools, the integration mechanism with the FL pipeline, and the challenges posed by the distributed and heterogeneous nature of FL environments.

Related papers

• Digital Twin-Assisted Federated Learning with Blockchain in Multi-tier Computing Systems [67.14406100332671]
  In Industry 4.0 systems, resource-constrained edge devices engage in frequent data interactions. This paper proposes a digital twin (DT) and federated digital twin (FL) scheme. The efficacy of our proposed cooperative interference-based FL process has been verified through numerical analysis.
  arXiv  Detail & Related papers  (2024-11-04T17:48:02Z)
• A Web-Based Solution for Federated Learning with LLM-Based Automation [34.756818299081736]
  Federated Learning (FL) offers a promising approach for collaborative machine learning across distributed devices. We develop a user-friendly web application supporting the federated averaging (FedAvg) algorithm. We explore intent-based automation in FL using a fine-tuned Language Model (LLM) trained on a tailored dataset.
  arXiv  Detail & Related papers  (2024-08-23T11:57:02Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. Experiments show that SpaFL improves accuracy while requiring much less communication and computing resources compared to sparse baselines.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [71.87335804334616]
  Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. The training process of Large Language Models (LLMs) generally incurs the update of significant parameters. This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
  arXiv  Detail & Related papers  (2023-10-23T16:37:59Z)
• Sample-Driven Federated Learning for Energy-Efficient and Real-Time IoT Sensing [22.968661040226756]
  We introduce an online reinforcement learning algorithm named Sample-driven Control for Federated Learning (SCFL) built on the Soft Actor-Critic (A2C) framework. SCFL enables the agent to dynamically adapt and find the global optima even in changing environments.
  arXiv  Detail & Related papers  (2023-10-11T13:50:28Z)
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence [83.58839320635956]
  Federated Learning (FL) can be used in mobile edge networks to train machine learning models in a distributed manner. Recent FL has been interpreted within a Model-Agnostic Meta-Learning (MAML) framework, which brings FL significant advantages in fast adaptation and convergence over heterogeneous datasets. This paper addresses how much benefit MAML brings to FL and how to maximize such benefit over mobile edge networks.
  arXiv  Detail & Related papers  (2023-03-23T02:42:10Z)
• Performance Optimization for Variable Bitwidth Federated Learning in Wireless Networks [103.22651843174471]
  This paper considers improving wireless communication and computation efficiency in federated learning (FL) via model quantization. In the proposed bitwidth FL scheme, edge devices train and transmit quantized versions of their local FL model parameters to a coordinating server, which aggregates them into a quantized global model and synchronizes the devices. We show that the FL training process can be described as a Markov decision process and propose a model-based reinforcement learning (RL) method to optimize action selection over iterations.
  arXiv  Detail & Related papers  (2022-09-21T08:52:51Z)
• Evaluation of Hyperparameter-Optimization Approaches in an Industrial Federated Learning System [0.2609784101826761]
  Federated Learning (FL) decouples model training from the need for direct access to the data. In this work, we investigated the impact of different hyperparameter optimization approaches in an FL system. We implemented these approaches based on grid search and Bayesian optimization and evaluated the algorithms on the MNIST data set and on the Internet of Things (IoT) sensor based industrial data set.
  arXiv  Detail & Related papers  (2021-10-15T17:01:40Z)
• Genetic CFL: Optimization of Hyper-Parameters in Clustered Federated Learning [4.710427287359642]
  Federated learning (FL) is a distributed model for deep learning that integrates client-server architecture, edge computing, and real-time intelligence. FL has the capability of revolutionizing machine learning (ML) but lacks in the practicality of implementation due to technological limitations, communication overhead, non-IID (independent and identically distributed) data, and privacy concerns. We propose a novel hybrid algorithm, namely genetic clustered FL (Genetic CFL), that clusters edge devices based on the training hyper- parameters and genetically modifies the parameters cluster-wise.
  arXiv  Detail & Related papers  (2021-07-15T10:16:05Z)
• Accelerating Federated Learning over Reliability-Agnostic Clients in Mobile Edge Computing Systems [15.923599062148135]
  Federated learning has emerged as a promising privacy-preserving approach to facilitating AI applications. It remains a big challenge to optimize the efficiency and effectiveness of FL when it is integrated with the MEC architecture. In this paper, a multi-layer federated learning protocol called HybridFL is designed for the MEC architecture.
  arXiv  Detail & Related papers  (2020-07-28T17:35:39Z)
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction [75.16836697734995]
  We propose a two-stage algorithm called Automatic Feature Interaction Selection (AutoFIS) AutoFIS can automatically identify important feature interactions for factorization models with computational cost just equivalent to training the target model to convergence. AutoFIS has been deployed onto the training platform of Huawei App Store recommendation service.
  arXiv  Detail & Related papers  (2020-03-25T06:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.