HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems
for HPO
- URL: http://arxiv.org/abs/2109.06716v1
- Date: Tue, 14 Sep 2021 14:28:51 GMT
- Title: HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems
for HPO
- Authors: Katharina Eggensperger, Philipp M\"uller, Neeratyoy Mallik, Matthias
Feurer, Ren\'e Sass, Aaron Klein, Noor Awad, Marius Lindauer, Frank Hutter
- Abstract summary: We propose HPOBench, which includes 7 existing and 5 new benchmark families, with in total more than 100 multi-fidelity benchmark problems.
HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers.
- Score: 30.89560505052524
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: To achieve peak predictive performance, hyperparameter optimization (HPO) is
a crucial component of machine learning and its applications. Over the last
years,the number of efficient algorithms and tools for HPO grew substantially.
At the same time, the community is still lacking realistic, diverse,
computationally cheap,and standardized benchmarks. This is especially the case
for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which
includes 7 existing and 5 new benchmark families, with in total more than 100
multi-fidelity benchmark problems. HPOBench allows to run this extendable set
of multi-fidelity HPO benchmarks in a reproducible way by isolating and
packaging the individual benchmarks in containers. It also provides surrogate
and tabular benchmarks for computationally affordable yet statistically sound
evaluations. To demonstrate the broad compatibility of HPOBench and its
usefulness, we conduct an exemplary large-scale study evaluating 6 well known
multi-fidelity HPO tools.
Related papers
- POGEMA: A Benchmark Platform for Cooperative Multi-Agent Navigation [76.67608003501479]
We introduce and specify an evaluation protocol defining a range of domain-related metrics computed on the basics of the primary evaluation indicators.
The results of such a comparison, which involves a variety of state-of-the-art MARL, search-based, and hybrid methods, are presented.
arXiv Detail & Related papers (2024-07-20T16:37:21Z) - Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks [40.8406006936244]
We introduce a Python package that facilitates efficient parallel HPO with zero-cost benchmarks.
Our approach calculates the exact return order based on the information stored in file system.
Our package can be installed via pip install mfhpo-simulator.
arXiv Detail & Related papers (2024-03-04T09:49:35Z) - FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization [50.12374973760274]
We propose and implement a benchmark suite FedHPO-B that incorporates comprehensive FL tasks, enables efficient function evaluations, and eases continuing extensions.
We also conduct extensive experiments based on FedHPO-B to benchmark a few HPO methods.
arXiv Detail & Related papers (2022-06-08T15:29:10Z) - A survey on multi-objective hyperparameter optimization algorithms for
Machine Learning [62.997667081978825]
This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms.
We distinguish between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both.
We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.
arXiv Detail & Related papers (2021-11-23T10:22:30Z) - LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark
Suite for Lasso [84.6451154376526]
LassoBench is a new benchmark suite tailored for an important open research topic in the Lasso community.
We evaluate 5 state-of-the-art HPO methods and 3 baselines, and demonstrate that Bayesian optimization, in particular, can improve over the methods commonly used for sparse regression.
arXiv Detail & Related papers (2021-11-04T12:05:09Z) - YAHPO Gym -- Design Criteria and a new Multifidelity Benchmark for
Hyperparameter Optimization [1.0718353079920009]
We present a new surrogate-based benchmark suite for multifidelity HPO methods consisting of 9 benchmark collections that constitute over 700 multifidelity HPO problems in total.
All our benchmarks also allow for querying of multiple optimization targets, enabling the benchmarking of multi-objective HPO.
arXiv Detail & Related papers (2021-09-08T14:16:31Z) - Hyperparameter Optimization: Foundations, Algorithms, Best Practices and
Open Challenges [5.139260825952818]
This paper reviews important HPO methods such as grid or random search, evolutionary algorithms, Bayesian optimization, Hyperband and racing.
It gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with ML pipelines, runtime improvements, and parallelization.
arXiv Detail & Related papers (2021-07-13T04:55:47Z) - HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on
OpenML [5.735035463793008]
We present HPO-B, a large-scale benchmark for comparing HPO algorithms.
Our benchmark is assembled and preprocessed from the OpenML repository.
We detail explicit experimental protocols, splits, and evaluation measures for comparing methods for both non-transfer and transfer learning HPO.
arXiv Detail & Related papers (2021-06-11T09:18:39Z) - The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games [67.47961797770249]
Multi-Agent PPO (MAPPO) is a multi-agent PPO variant which adopts a centralized value function.
We show that MAPPO achieves performance comparable to the state-of-the-art in three popular multi-agent testbeds.
arXiv Detail & Related papers (2021-03-02T18:59:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.