HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

• URL: http://arxiv.org/abs/2109.06716v1
• Date: Tue, 14 Sep 2021 14:28:51 GMT
• Title: HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Authors: Katharina Eggensperger, Philipp M\"uller, Neeratyoy Mallik, Matthias Feurer, Ren\'e Sass, Aaron Klein, Noor Awad, Marius Lindauer, Frank Hutter
• Abstract summary: We propose HPOBench, which includes 7 existing and 5 new benchmark families, with in total more than 100 multi-fidelity benchmark problems. HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers.
• Score: 30.89560505052524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years,the number of efficient algorithms and tools for HPO grew substantially. At the same time, the community is still lacking realistic, diverse, computationally cheap,and standardized benchmarks. This is especially the case for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which includes 7 existing and 5 new benchmark families, with in total more than 100 multi-fidelity benchmark problems. HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers. It also provides surrogate and tabular benchmarks for computationally affordable yet statistically sound evaluations. To demonstrate the broad compatibility of HPOBench and its usefulness, we conduct an exemplary large-scale study evaluating 6 well known multi-fidelity HPO tools.

Related papers

• Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction [71.81851971324187]
  This work introduces Hierarchical Preference Optimization (HPO), a novel approach to hierarchical reinforcement learning (HRL) HPO addresses non-stationarity and infeasible subgoal generation issues when solving complex robotic control tasks. Experiments on challenging robotic navigation and manipulation tasks demonstrate impressive performance of HPO, where it shows an improvement of up to 35% over the baselines.
  arXiv  Detail & Related papers  (2024-11-01T04:58:40Z)
• The Perfect Blend: Redefining RLHF with Mixture of Judges [68.58426626501883]
  Reinforcement learning from human feedback (RLHF) has become the leading approach for fine-tuning large language models (LLM) Applying RLHF for MTL currently requires careful tuning of the weights for reward model and data combinations. We introduce a novel post-training paradigm which we called Constrained Generative Policy Optimization (CGPO)
  arXiv  Detail & Related papers  (2024-09-30T15:06:53Z)
• Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks [40.8406006936244]
  We introduce a Python package that facilitates efficient parallel HPO with zero-cost benchmarks. Our approach calculates the exact return order based on the information stored in file system. Our package can be installed via pip install mfhpo-simulator.
  arXiv  Detail & Related papers  (2024-03-04T09:49:35Z)
• FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization [50.12374973760274]
  We propose and implement a benchmark suite FedHPO-B that incorporates comprehensive FL tasks, enables efficient function evaluations, and eases continuing extensions. We also conduct extensive experiments based on FedHPO-B to benchmark a few HPO methods.
  arXiv  Detail & Related papers  (2022-06-08T15:29:10Z)
• LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso [84.6451154376526]
  LassoBench is a new benchmark suite tailored for an important open research topic in the Lasso community. We evaluate 5 state-of-the-art HPO methods and 3 baselines, and demonstrate that Bayesian optimization, in particular, can improve over the methods commonly used for sparse regression.
  arXiv  Detail & Related papers  (2021-11-04T12:05:09Z)
• YAHPO Gym -- Design Criteria and a new Multifidelity Benchmark for Hyperparameter Optimization [1.0718353079920009]
  We present a new surrogate-based benchmark suite for multifidelity HPO methods consisting of 9 benchmark collections that constitute over 700 multifidelity HPO problems in total. All our benchmarks also allow for querying of multiple optimization targets, enabling the benchmarking of multi-objective HPO.
  arXiv  Detail & Related papers  (2021-09-08T14:16:31Z)
• Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges [5.139260825952818]
  This paper reviews important HPO methods such as grid or random search, evolutionary algorithms, Bayesian optimization, Hyperband and racing. It gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with ML pipelines, runtime improvements, and parallelization.
  arXiv  Detail & Related papers  (2021-07-13T04:55:47Z)
• HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML [5.735035463793008]
  We present HPO-B, a large-scale benchmark for comparing HPO algorithms. Our benchmark is assembled and preprocessed from the OpenML repository. We detail explicit experimental protocols, splits, and evaluation measures for comparing methods for both non-transfer and transfer learning HPO.
  arXiv  Detail & Related papers  (2021-06-11T09:18:39Z)
• The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games [67.47961797770249]
  Multi-Agent PPO (MAPPO) is a multi-agent PPO variant which adopts a centralized value function. We show that MAPPO achieves performance comparable to the state-of-the-art in three popular multi-agent testbeds.
  arXiv  Detail & Related papers  (2021-03-02T18:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.