FastBO: Fast HPO and NAS with Adaptive Fidelity Identification

• URL: http://arxiv.org/abs/2409.00584v1
• Date: Sun, 1 Sep 2024 02:40:04 GMT
• Title: FastBO: Fast HPO and NAS with Adaptive Fidelity Identification
• Authors: Jiantong Jiang, Ajmal Mian,
• Abstract summary: We propose a multi-fidelity BO method named FastBO, which adaptively decides the fidelity for each configuration and efficiently offers strong performance. We also show that our adaptive fidelity identification strategy provides a way to extend any single-fidelity method to the multi-fidelity setting.
• Score: 29.594900930334216
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyperparameter optimization (HPO) and neural architecture search (NAS) are powerful in attaining state-of-the-art machine learning models, with Bayesian optimization (BO) standing out as a mainstream method. Extending BO into the multi-fidelity setting has been an emerging research topic, but faces the challenge of determining an appropriate fidelity for each hyperparameter configuration to fit the surrogate model. To tackle the challenge, we propose a multi-fidelity BO method named FastBO, which adaptively decides the fidelity for each configuration and efficiently offers strong performance. The advantages are achieved based on the novel concepts of efficient point and saturation point for each configuration.We also show that our adaptive fidelity identification strategy provides a way to extend any single-fidelity method to the multi-fidelity setting, highlighting its generality and applicability.

Related papers

• FADAS: Towards Federated Adaptive Asynchronous Optimization [56.09666452175333]
  Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning. This paper introduces federated adaptive asynchronous optimization, named FADAS, a novel method that incorporates asynchronous updates into adaptive federated optimization with provable guarantees. We rigorously establish the convergence rate of the proposed algorithms and empirical results demonstrate the superior performance of FADAS over other asynchronous FL baselines.
  arXiv  Detail & Related papers  (2024-07-25T20:02:57Z)
• Decoding-Time Language Model Alignment with Multiple Objectives [116.42095026960598]
  Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives. Here, we propose $textbfmulti-objective decoding (MOD)$, a decoding-time algorithm that outputs the next token from a linear combination of predictions. We show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method.
  arXiv  Detail & Related papers  (2024-06-27T02:46:30Z)
• Adaptive Preference Scaling for Reinforcement Learning with Human Feedback [103.36048042664768]
  Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values. We propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO) Our method is versatile and can be readily adapted to various preference optimization frameworks.
  arXiv  Detail & Related papers  (2024-06-04T20:33:22Z)
• FlexHB: a More Efficient and Flexible Framework for Hyperparameter Optimization [4.127081624438282]
  We propose FlexHB, a new method pushing multi-fidelity BO to the limit and re-designing a framework for early stopping with Successive Halving(SH) Our method achieves superior efficiency and outperforms other methods on various HPO tasks.
  arXiv  Detail & Related papers  (2024-02-21T09:18:59Z)
• BOtied: Multi-objective Bayesian optimization with tied multivariate ranks [33.414682601242006]
  In this paper, we show a natural connection between non-dominated solutions and the extreme quantile of the joint cumulative distribution function. Motivated by this link, we propose the Pareto-compliant CDF indicator and the associated acquisition function, BOtied. Our experiments on a variety of synthetic and real-world problems demonstrate that BOtied outperforms state-of-the-art MOBO acquisition functions.
  arXiv  Detail & Related papers  (2023-06-01T04:50:06Z)
• MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization [30.54386890506418]
  MO-DEHB is an effective and flexible multi-objective (MO) that extends the recent evolutionary Hyperband method DEHB. A comparative study against state-of-the-art MOs demonstrates that MO-DEHB clearly achieves the best performance across our 15 benchmarks.
  arXiv  Detail & Related papers  (2023-05-08T06:53:40Z)
• Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures [68.91874045918112]
  adapter-ALBERT is an efficient model optimization for maximal data reuse across different tasks. We demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator.
  arXiv  Detail & Related papers  (2023-03-25T14:40:59Z)
• Accelerated Federated Learning with Decoupled Adaptive Optimization [53.230515878096426]
  federated learning (FL) framework enables clients to collaboratively learn a shared model while keeping privacy of training data on clients. Recently, many iterations efforts have been made to generalize centralized adaptive optimization methods, such as SGDM, Adam, AdaGrad, etc., to federated settings. This work aims to develop novel adaptive optimization methods for FL from the perspective of dynamics of ordinary differential equations (ODEs)
  arXiv  Detail & Related papers  (2022-07-14T22:46:43Z)
• Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks [17.370056935194786]
  We propose Batch Multi-fidelity Bayesian Optimization with Deep Auto-Regressive Networks (BMBO-DARN) We use a set of Bayesian neural networks to construct a fully auto-regressive model, which is expressive enough to capture strong yet complex relationships across all fidelities. We develop a simple yet efficient batch querying method, without any search over fidelities.
  arXiv  Detail & Related papers  (2021-06-18T02:55:48Z)
• Optimization-Inspired Learning with Architecture Augmentations and Control Mechanisms for Low-Level Vision [74.9260745577362]
  This paper proposes a unified optimization-inspired learning framework to aggregate Generative, Discriminative, and Corrective (GDC) principles. We construct three propagative modules to effectively solve the optimization models with flexible combinations. Experiments across varied low-level vision tasks validate the efficacy and adaptability of GDC.
  arXiv  Detail & Related papers  (2020-12-10T03:24:53Z)
• Multi-Fidelity Bayesian Optimization via Deep Neural Networks [19.699020509495437]
  In many applications, the objective function can be evaluated at multiple fidelities to enable a trade-off between the cost and accuracy. We propose Deep Neural Network Multi-Fidelity Bayesian Optimization (DNN-MFBO) that can flexibly capture all kinds of complicated relationships between the fidelities. We show the advantages of our method in both synthetic benchmark datasets and real-world applications in engineering design.
  arXiv  Detail & Related papers  (2020-07-06T23:28:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.