Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020

• URL: http://arxiv.org/abs/2104.10201v1
• Date: Tue, 20 Apr 2021 18:44:59 GMT
• Title: Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
• Abstract summary: This paper presents the results and insights from the black-box optimization challenge at NeurIPS 2020. This was the first black-box optimization challenge with a machine learning emphasis.
• Score: 10.782941183498851
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.

Related papers

• An Open-Source Training Dataset for Foundation Models for Black-box Optimization [6.259657082419056]
  We introduce BBO-Pile, the first open-source dataset comprising over 500K optimization trajectories across 30 different black-boxes.<n>We train a family of foundation models at multiple scales, ranging from 2M to 80M parameters.<n>Our results demonstrate that large-scale pre-training is a viable and effective approach to imitate black-box optimization methods.
  arXiv  Detail & Related papers  (2026-05-22T09:27:59Z)
• ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimization [31.894110383242566]
  We present ZeroShot, a general-purpose, pretrained model for continuous black-box optimization tasks ranging from 2D to 20D.<n>Our approach leverages offline reinforcement learning on large-scale optimization tasks collected from 12 BO variants.
  arXiv  Detail & Related papers  (2025-10-03T14:33:23Z)
• BOOST: Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique [1.0499611180329804]
  We propose BOOST, a novel framework that automates the selection of kernel-acquisition pairs.<n>BOOST predicts the performance of various kernel-acquisition pairs and identifies the most promising pair before committing to expensive evaluations.
  arXiv  Detail & Related papers  (2025-08-04T12:08:12Z)
• Efficient optimization of expensive black-box simulators via marginal means, with application to neutrino detector design [1.5749416770494706]
  We propose a new Black-box Optimization via Marginal Means (BOMM) approach.<n>BOMM uses a new estimator of a global $mathbfx*$ that can be efficiently inferred with limited runs in high dimensions.<n>We show that BOMM is consistent for optimization, but also has an optimization rate that tempers the ''curse-of-dimensionality'' faced by existing methods.
  arXiv  Detail & Related papers  (2025-08-03T16:44:05Z)
• Discovering Preference Optimization Algorithms with and for Large Language Models [50.843710797024805]
  offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. We perform objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Experiments demonstrate the state-of-the-art performance of DiscoPOP, a novel algorithm that adaptively blends logistic and exponential losses.
  arXiv  Detail & Related papers  (2024-06-12T16:58:41Z)
• End-to-End Learning for Fair Multiobjective Optimization Under Uncertainty [55.04219793298687]
  The Predict-Then-Forecast (PtO) paradigm in machine learning aims to maximize downstream decision quality. This paper extends the PtO methodology to optimization problems with nondifferentiable Ordered Weighted Averaging (OWA) objectives. It shows how optimization of OWA functions can be effectively integrated with parametric prediction for fair and robust optimization under uncertainty.
  arXiv  Detail & Related papers  (2024-02-12T16:33:35Z)
• MADA: Meta-Adaptive Optimizers through hyper-gradient Descent [73.1383658672682]
  We introduce Meta-Adaptives (MADA), a unified framework that can generalize several known convergences and dynamically learn the most suitable one during training. We empirically compare MADA to other populars on vision and language tasks, and find that MADA consistently outperforms Adam and other populars. We also propose AVGrad, a modification of AMS that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization.
  arXiv  Detail & Related papers  (2024-01-17T00:16:46Z)
• CoRe Optimizer: An All-in-One Solution for Machine Learning [0.0]
  Continuously resilient convergence (CoRe) shown superior performance compared to other state-of-the-art first-order gradient-based convergence algorithms. CoRe yields best or competitive performance in every investigated application.
  arXiv  Detail & Related papers  (2023-07-28T16:48:42Z)
• VeLO: Training Versatile Learned Optimizers by Scaling Up [67.90237498659397]
  We leverage the same scaling approach behind the success of deep learning to learn versatiles. We train an ingest for deep learning which is itself a small neural network that ingests and outputs parameter updates. We open source our learned, meta-training code, the associated train test data, and an extensive benchmark suite with baselines at velo-code.io.
  arXiv  Detail & Related papers  (2022-11-17T18:39:07Z)
• Pre-training helps Bayesian optimization too [49.28382118032923]
  We seek an alternative practice for setting functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. Our results show that our method is able to locate good hyper parameters at least 3 times more efficiently than the best competing methods.
  arXiv  Detail & Related papers  (2022-07-07T04:42:54Z)
• Learning How to Optimize Black-Box Functions With Extreme Limits on the Number of Function Evaluations [3.0969191504482243]
  We consider black-box optimization in which only an extremely limited number of function evaluations, on the order of around 100, are affordable. We propose an original method that uses established approaches to propose a set of points for each batch and then down-selects from these candidate points to the number of trials that can be run in parallel. We achieve an average reduction of 50% of normalized cost, which is a highly significant improvement in performance.
  arXiv  Detail & Related papers  (2021-03-18T15:30:15Z)
• Bayesian Optimization for Selecting Efficient Machine Learning Models [53.202224677485525]
  We present a unified Bayesian Optimization framework for jointly optimizing models for both prediction effectiveness and training efficiency. Experiments on model selection for recommendation tasks indicate models selected this way significantly improves model training efficiency.
  arXiv  Detail & Related papers  (2020-08-02T02:56:30Z)
• Towards Automatic Bayesian Optimization: A first step involving acquisition functions [0.0]
  Bayesian optimization is the state of the art technique for the optimization of black boxes, i.e., functions where we do not have access to their analytical expression. We propose a first attempt over automatic bayesian optimization by exploring several techniques that automatically tune the acquisition function.
  arXiv  Detail & Related papers  (2020-03-21T12:22:45Z)
• Scalable Constrained Bayesian Optimization [10.820024633762596]
  The global optimization of a high-dimensional black-box function under black-box constraints is a pervasive task in machine learning, control, and the scientific community. We propose the scalable constrained Bayesian optimization (SCBO) algorithm that overcomes the above challenges and pushes the state-the-art.
  arXiv  Detail & Related papers  (2020-02-20T01:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.