DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization

• URL: http://arxiv.org/abs/2105.09821v1
• Date: Thu, 20 May 2021 15:13:30 GMT
• Title: DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization
• Authors: Noor Awad, Neeratyoy Mallik, Frank Hutter
• Abstract summary: We present a new HPO method which we call DEHB. It achieves strong performance far more robustly than all previous HPO methods. It is also efficient in computational time, conceptually simple and easy to implement, positioning it well to become a new default HPO method.
• Score: 33.80873355096445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever. Here, we combine the advantages of the popular bandit-based HPO method Hyperband (HB) and the evolutionary search approach of Differential Evolution (DE) to yield a new HPO method which we call DEHB. Comprehensive results on a very broad range of HPO problems, as well as a wide range of tabular benchmarks from neural architecture search, demonstrate that DEHB achieves strong performance far more robustly than all previous HPO methods we are aware of, especially for high-dimensional problems with discrete input dimensions. For example, DEHB is up to 1000x faster than random search. It is also efficient in computational time, conceptually simple and easy to implement, positioning it well to become a new default HPO method.

Related papers

• Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction [71.81851971324187]
  This work introduces Hierarchical Preference Optimization (HPO), a novel approach to hierarchical reinforcement learning (HRL) HPO addresses non-stationarity and infeasible subgoal generation issues when solving complex robotic control tasks. Experiments on challenging robotic navigation and manipulation tasks demonstrate impressive performance of HPO, where it shows an improvement of up to 35% over the baselines.
  arXiv  Detail & Related papers  (2024-11-01T04:58:40Z)
• HomOpt: A Homotopy-Based Hyperparameter Optimization Method [10.11271414863925]
  We propose HomOpt, a data-driven approach based on a generalized additive model (GAM) surrogate combined with homotopy optimization. We show how HomOpt can boost the performance and effectiveness of any given method with faster convergence to the optimum on continuous discrete, and categorical domain spaces.
  arXiv  Detail & Related papers  (2023-08-07T06:01:50Z)
• Enhancing Explainability of Hyperparameter Optimization via Bayesian Algorithm Execution [13.037647287689438]
  We study the combination of HPO with interpretable machine learning (IML) methods such as partial dependence plots. We propose a modified HPO method which efficiently searches for optimum global predictive performance. Our method returns more reliable explanations of the underlying black-box without a loss of optimization performance.
  arXiv  Detail & Related papers  (2022-06-11T07:12:04Z)
• FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization [50.12374973760274]
  We propose and implement a benchmark suite FedHPO-B that incorporates comprehensive FL tasks, enables efficient function evaluations, and eases continuing extensions. We also conduct extensive experiments based on FedHPO-B to benchmark a few HPO methods.
  arXiv  Detail & Related papers  (2022-06-08T15:29:10Z)
• Towards Learning Universal Hyperparameter Optimizers with Transformers [57.35920571605559]
  We introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction. Our experiments demonstrate that the OptFormer can imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates.
  arXiv  Detail & Related papers  (2022-05-26T12:51:32Z)
• Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers [3.729201909920989]
  We present a principled approach to automated benchmark-driven algorithm design applied to multi parameter HPO (MF-HPO) First, we formalize a rich space of MF-HPO candidates that includes, but is not limited to common HPO algorithms, and then present a framework covering this space. We challenge whether the found design choices are necessary or could be replaced by more naive and simpler ones by performing an ablation analysis.
  arXiv  Detail & Related papers  (2021-11-29T18:02:56Z)
• A survey on multi-objective hyperparameter optimization algorithms for Machine Learning [62.997667081978825]
  This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms. We distinguish between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.
  arXiv  Detail & Related papers  (2021-11-23T10:22:30Z)
• Genealogical Population-Based Training for Hyperparameter Optimization [1.0514231683620516]
  We experimentally demonstrate that our method cuts down by 2 to 3 times the computational cost required. Our method is search-algorithm so that the inner search routine can be any search algorithm like TPE, GP, CMA or random search.
  arXiv  Detail & Related papers  (2021-09-30T08:49:41Z)
• Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges [5.139260825952818]
  This paper reviews important HPO methods such as grid or random search, evolutionary algorithms, Bayesian optimization, Hyperband and racing. It gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with ML pipelines, runtime improvements, and parallelization.
  arXiv  Detail & Related papers  (2021-07-13T04:55:47Z)
• HyperSTAR: Task-Aware Hyperparameters for Deep Networks [52.50861379908611]
  HyperSTAR is a task-aware method to warm-start HPO for deep neural networks. It learns a dataset (task) representation along with the performance predictor directly from raw images. It evaluates 50% less configurations to achieve the best performance compared to existing methods.
  arXiv  Detail & Related papers  (2020-05-21T08:56:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.