Surrogate modeling for Bayesian optimization beyond a single Gaussian process

• URL: http://arxiv.org/abs/2205.14090v1
• Date: Fri, 27 May 2022 16:43:10 GMT
• Title: Surrogate modeling for Bayesian optimization beyond a single Gaussian process
• Authors: Qin Lu, Konstantinos D. Polyzos, Bingcong Li, Georgios B. Giannakis
• Abstract summary: We propose a novel Bayesian surrogate model to balance exploration with exploitation of the search space. To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model. To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret.
• Score: 62.294228304646516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bayesian optimization (BO) has well-documented merits for optimizing black-box functions with an expensive evaluation cost. Such functions emerge in applications as diverse as hyperparameter tuning, drug discovery, and robotics. BO hinges on a Bayesian surrogate model to sequentially select query points so as to balance exploration with exploitation of the search space. Most existing works rely on a single Gaussian process (GP) based surrogate model, where the kernel function form is typically preselected using domain knowledge. To bypass such a design process, this paper leverages an ensemble (E) of GPs to adaptively select the surrogate model fit on-the-fly, yielding a GP mixture posterior with enhanced expressiveness for the sought function. Acquisition of the next evaluation input using this EGP-based function posterior is then enabled by Thompson sampling (TS) that requires no additional design parameters. To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model. The novel EGP-TS readily accommodates parallel operation. To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret for both sequential and parallel settings. Tests on synthetic functions and real-world applications showcase the merits of the proposed method.

Related papers

• Optimizing Posterior Samples for Bayesian Optimization via Rootfinding [2.94944680995069]
  We introduce an efficient global optimization strategy for posterior samples based on global rootfinding. We demonstrate remarkable improvement in both inner- and outer-loop optimization. We also propose a sample-average formulation of GP-TS, which has a parameter to explicitly control exploitation.
  arXiv  Detail & Related papers  (2024-10-29T17:57:16Z)
• Poisson Process for Bayesian Optimization [126.51200593377739]
  We propose a ranking-based surrogate model based on the Poisson process and introduce an efficient BO framework, namely Poisson Process Bayesian Optimization (PoPBO) Compared to the classic GP-BO method, our PoPBO has lower costs and better robustness to noise, which is verified by abundant experiments.
  arXiv  Detail & Related papers  (2024-02-05T02:54:50Z)
• Simulation Based Bayesian Optimization [0.6526824510982799]
  This paper introduces Simulation Based Bayesian Optimization (SBBO) as a novel approach to optimizing acquisition functions. SBBO allows the use of surrogate models tailored for spaces with discrete variables. We demonstrate empirically the effectiveness of SBBO method using various choices of surrogate models.
  arXiv  Detail & Related papers  (2024-01-19T16:56:11Z)
• Bayesian Kernelized Tensor Factorization as Surrogate for Bayesian Optimization [13.896697187967545]
  Kernel optimization (BO) primarily uses Gaussian processes (GP) as the key surrogate model. In this paper, we propose to use Bayesian Factorization (BKTF) as a new surrogate model -- for BO in a $D$-dimensional product space. BKTF offers a flexible and highly effective approach for characterizing complex functions with uncertainty quantification.
  arXiv  Detail & Related papers  (2023-02-28T12:00:21Z)
• Scalable Bayesian Optimization Using Vecchia Approximations of Gaussian Processes [0.0]
  We adapt the Vecchia approximation, a popular GP approximation from spatial statistics, to enable scalable high-dimensional Bayesian optimization. We focus on the use of our warped Vecchia GP in trust-region Bayesian optimization via Thompson sampling.
  arXiv  Detail & Related papers  (2022-03-02T23:55:14Z)
• Non-Gaussian Gaussian Processes for Few-Shot Regression [71.33730039795921]
  We propose an invertible ODE-based mapping that operates on each component of the random variable vectors and shares the parameters across all of them. NGGPs outperform the competing state-of-the-art approaches on a diversified set of benchmarks and applications.
  arXiv  Detail & Related papers  (2021-10-26T10:45:25Z)
• Gaussian Process Sampling and Optimization with Approximate Upper and Lower Bounds [43.70206216468687]
  Many functions have approximately-known upper and/or lower bounds, potentially aiding the modeling of such functions. We introduce Gaussian process models for functions where such bounds are (approximately) known. That is, we transform a GP model satisfying the given bounds, and then sample and weight functions from its posterior.
  arXiv  Detail & Related papers  (2021-10-22T22:35:57Z)
• Incremental Ensemble Gaussian Processes [53.3291389385672]
  We propose an incremental ensemble (IE-) GP framework, where an EGP meta-learner employs an it ensemble of GP learners, each having a unique kernel belonging to a prescribed kernel dictionary. With each GP expert leveraging the random feature-based approximation to perform online prediction and model update with it scalability, the EGP meta-learner capitalizes on data-adaptive weights to synthesize the per-expert predictions. The novel IE-GP is generalized to accommodate time-varying functions by modeling structured dynamics at the EGP meta-learner and within each GP learner.
  arXiv  Detail & Related papers  (2021-10-13T15:11:25Z)
• Likelihood-Free Inference with Deep Gaussian Processes [70.74203794847344]
  Surrogate models have been successfully used in likelihood-free inference to decrease the number of simulator evaluations. We propose a Deep Gaussian Process (DGP) surrogate model that can handle more irregularly behaved target distributions. Our experiments show how DGPs can outperform GPs on objective functions with multimodal distributions and maintain a comparable performance in unimodal cases.
  arXiv  Detail & Related papers  (2020-06-18T14:24:05Z)
• Randomised Gaussian Process Upper Confidence Bound for Bayesian Optimisation [60.93091603232817]
  We develop a modified Gaussian process upper confidence bound (GP-UCB) acquisition function. This is done by sampling the exploration-exploitation trade-off parameter from a distribution. We prove that this allows the expected trade-off parameter to be altered to better suit the problem without compromising a bound on the function's Bayesian regret.
  arXiv  Detail & Related papers  (2020-06-08T00:28:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.