Iterative Deepening Hyperband

• URL: http://arxiv.org/abs/2302.00511v1
• Date: Wed, 1 Feb 2023 15:33:51 GMT
• Title: Iterative Deepening Hyperband
• Authors: Jasmin Brandt, Marcel Wever, Dimitrios Iliadis, Viktor Bengs, Eyke H\"ullermeier
• Abstract summary: We show that incremental variants of Hyperband satisfy theoretical guarantees qualitatively similar to those for the original Hyperband with the "right" budget. We demonstrate their practical utility in experiments with benchmark data sets.
• Score: 8.257520009686239
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperparameter optimization (HPO) is concerned with the automated search for the most appropriate hyperparameter configuration (HPC) of a parameterized machine learning algorithm. A state-of-the-art HPO method is Hyperband, which, however, has its own parameters that influence its performance. One of these parameters, the maximal budget, is especially problematic: If chosen too small, the budget needs to be increased in hindsight and, as Hyperband is not incremental by design, the entire algorithm must be re-run. This is not only costly but also comes with a loss of valuable knowledge already accumulated. In this paper, we propose incremental variants of Hyperband that eliminate these drawbacks, and show that these variants satisfy theoretical guarantees qualitatively similar to those for the original Hyperband with the "right" budget. Moreover, we demonstrate their practical utility in experiments with benchmark data sets.

Related papers

• Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning [51.6191275658441]
  We introduce emphutility in the freeze-thaw framework, a function describing the trade-off between the cost and performance.<n>We validate our algorithm on established multi-fidelity HPO benchmarks and show that it outperforms all the previous freeze-thaw BO and transfer-BO baselines.
  arXiv  Detail & Related papers  (2025-10-24T12:03:57Z)
• Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW [11.681640186200951]
  We present a set of practical and performant hyper parameter lists for NAdamW. Our best NAdamW hyper parameter list performs well on AlgoPerf held-out workloads not used to construct it. It also outperforms basic learning rate/weight decay sweeps and an off-the-shelf Bayesian optimization tool when restricted to the same budget.
  arXiv  Detail & Related papers  (2025-03-06T00:14:50Z)
• PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning [49.92394599459274]
  We propose PriorBand, an HPO algorithm tailored to Deep Learning (DL) pipelines. We show its robustness across a range of DL benchmarks and show its gains under informative expert input and against poor expert beliefs.
  arXiv  Detail & Related papers  (2023-06-21T16:26:14Z)
• Online Continuous Hyperparameter Optimization for Generalized Linear Contextual Bandits [55.03293214439741]
  In contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience. We propose the first online continuous hyperparameter tuning framework for contextual bandits. We show that it could achieve a sublinear regret in theory and performs consistently better than all existing methods on both synthetic and real datasets.
  arXiv  Detail & Related papers  (2023-02-18T23:31:20Z)
• A Lipschitz Bandits Approach for Continuous Hyperparameter Optimization [6.572589601317779]
  BLiE is a Lipschitz-bandit-based HPO algorithm that only assumes Lipschitz continuity of the objective function. Empirically, we demonstrate that BLiE outperforms the state-of-the-art HPO algorithms on benchmark tasks.
  arXiv  Detail & Related papers  (2023-02-03T04:30:17Z)
• A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization [57.450449884166346]
  We propose an adaptive HPO method to account for the privacy cost of HPO. We obtain state-of-the-art performance on 22 benchmark tasks, across computer vision and natural language processing, across pretraining and finetuning.
  arXiv  Detail & Related papers  (2022-12-08T18:56:37Z)
• HyP-ABC: A Novel Automated Hyper-Parameter Tuning Algorithm Using Evolutionary Optimization [1.6114012813668934]
  We propose HyP-ABC, an automatic hybrid hyper-parameter optimization algorithm using the modified artificial bee colony approach. Compared to the state-of-the-art techniques, HyP-ABC is more efficient and has a limited number of parameters to be tuned.
  arXiv  Detail & Related papers  (2021-09-11T16:45:39Z)
• Amortized Auto-Tuning: Cost-Efficient Transfer Optimization for Hyperparameter Recommendation [83.85021205445662]
  We propose an instantiation--amortized auto-tuning (AT2) to speed up tuning of machine learning models. We conduct a thorough analysis of the multi-task multi-fidelity Bayesian optimization framework, which leads to the best instantiation--amortized auto-tuning (AT2)
  arXiv  Detail & Related papers  (2021-06-17T00:01:18Z)
• Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm [97.66038345864095]
  We propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG) Specifically, we first formulate hyperparameter optimization as an A-based constrained optimization problem. Then, we use the average zeroth-order hyper-gradients to update hyper parameters.
  arXiv  Detail & Related papers  (2021-02-17T21:03:05Z)
• Cost-Efficient Online Hyperparameter Optimization [94.60924644778558]
  We propose an online HPO algorithm that reaches human expert-level performance within a single run of the experiment. Our proposed online HPO algorithm reaches human expert-level performance within a single run of the experiment, while incurring only modest computational overhead compared to regular training.
  arXiv  Detail & Related papers  (2021-01-17T04:55:30Z)
• Frugal Optimization for Cost-related Hyperparameters [43.599155206275306]
  We develop a new cost-frugal HPO solution for machine learning algorithms. We prove a convergence rate of $O(fracsqrtdsqrtK)$ and an $O(depsilon-2)$-approximation guarantee on the total cost. We provide strong empirical results in comparison with state-of-the-art HPO methods on large AutoML benchmarks.
  arXiv  Detail & Related papers  (2020-05-04T15:40:44Z)
• Automatic Hyper-Parameter Optimization Based on Mapping Discovery from Data to Hyper-Parameters [3.37314595161109]
  We propose an efficient automatic parameter optimization approach, which is based on the mapping from data to the corresponding hyper- parameters. We show that the proposed approaches outperform the state-of-the-art apporaches significantly.
  arXiv  Detail & Related papers  (2020-03-03T19:26:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.