Simplifying Bayesian Optimization Via In-Context Direct Optimum Sampling

• URL: http://arxiv.org/abs/2505.23913v1
• Date: Thu, 29 May 2025 18:07:36 GMT
• Title: Simplifying Bayesian Optimization Via In-Context Direct Optimum Sampling
• Authors: Gustavo Sutter Pessurno de Carvalho, Mohammed Abdulrahman, Hao Wang, Sriram Ganapathi Subramanian, Marc St-Aubin, Sharon O'Sullivan, Lawrence Wan, Luis Ricardez-Sandoval, Pascal Poupart, Agustinus Kristiadi,
• Abstract summary: We propose a completely in-context zero-shot solution for BO that does not require surrogate fitting or acquisition function optimization.<n>This is done by using a pre-trained context model to directly sample from the posterior over the optimum point.<n>We achieve an efficiency gain of more than 35x in terms of wall-clock time when compared with process-based BO.
• Score: 18.852567298468742
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The optimization of expensive black-box functions is ubiquitous in science and engineering. A common solution to this problem is Bayesian optimization (BO), which is generally comprised of two components: (i) a surrogate model and (ii) an acquisition function, which generally require expensive re-training and optimization steps at each iteration, respectively. Although recent work enabled in-context surrogate models that do not require re-training, virtually all existing BO methods still require acquisition function maximization to select the next observation, which introduces many knobs to tune, such as Monte Carlo samplers and multi-start optimizers. In this work, we propose a completely in-context, zero-shot solution for BO that does not require surrogate fitting or acquisition function optimization. This is done by using a pre-trained deep generative model to directly sample from the posterior over the optimum point. We show that this process is equivalent to Thompson sampling and demonstrate the capabilities and cost-effectiveness of our foundation model on a suite of real-world benchmarks. We achieve an efficiency gain of more than 35x in terms of wall-clock time when compared with Gaussian process-based BO, enabling efficient parallel and distributed BO, e.g., for high-throughput optimization.

Related papers

• Batched Energy-Entropy acquisition for Bayesian Optimization [0.0]
  We propose a statistical physics inspired acquisition function for BO with Gaussian processes that can handle batches. BEEBO enables tight control of the explore-exploit trade-off of the optimization process and generalizes to heteroskedastic black-box problems.
  arXiv  Detail & Related papers  (2024-10-11T13:39:47Z)
• Cost-Sensitive Multi-Fidelity Bayesian Optimization with Transfer of Learning Curve Extrapolation [55.75188191403343]
  We introduce utility, which is a function predefined by each user and describes the trade-off between cost and performance of BO. We validate our algorithm on various LC datasets and found it outperform all the previous multi-fidelity BO and transfer-BO baselines we consider.
  arXiv  Detail & Related papers  (2024-05-28T07:38:39Z)
• Poisson Process for Bayesian Optimization [126.51200593377739]
  We propose a ranking-based surrogate model based on the Poisson process and introduce an efficient BO framework, namely Poisson Process Bayesian Optimization (PoPBO) Compared to the classic GP-BO method, our PoPBO has lower costs and better robustness to noise, which is verified by abundant experiments.
  arXiv  Detail & Related papers  (2024-02-05T02:54:50Z)
• qPOTS: Efficient batch multiobjective Bayesian optimization via Pareto optimal Thompson sampling [0.0]
  A sample-efficient approach to solving multiobjective optimization is via process oracle (GP) surrogates and MOBOOTS$.<n>We propose a Thompson sampling (TS) based approach ($qtextttPOTS$)<n>$qtextttPOTS$ solves a cheap multiobjective optimization on the GP posteriors with evolutionary approaches.
  arXiv  Detail & Related papers  (2023-10-24T12:35:15Z)
• Predictive Modeling through Hyper-Bayesian Optimization [60.586813904500595]
  We propose a novel way of integrating model selection and BO for the single goal of reaching the function optima faster. The algorithm moves back and forth between BO in the model space and BO in the function space, where the goodness of the recommended model is captured. In addition to improved sample efficiency, the framework outputs information about the black-box function.
  arXiv  Detail & Related papers  (2023-08-01T04:46:58Z)
• Pre-training helps Bayesian optimization too [49.28382118032923]
  We seek an alternative practice for setting functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. Our results show that our method is able to locate good hyper parameters at least 3 times more efficiently than the best competing methods.
  arXiv  Detail & Related papers  (2022-07-07T04:42:54Z)
• Surrogate modeling for Bayesian optimization beyond a single Gaussian process [62.294228304646516]
  We propose a novel Bayesian surrogate model to balance exploration with exploitation of the search space. To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model. To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret.
  arXiv  Detail & Related papers  (2022-05-27T16:43:10Z)
• Bayesian Optimization over Permutation Spaces [30.650753803587794]
  We propose and evaluate two algorithms for BO over Permutation Spaces (BOPS) We theoretically analyze the performance of BOPS-T to show that their regret grows sub-linearly. Our experiments on multiple synthetic and real-world benchmarks show that both BOPS-T and BOPS-H perform better than the state-of-the-art BO algorithm for spaces.
  arXiv  Detail & Related papers  (2021-12-02T08:20:50Z)
• LinEasyBO: Scalable Bayesian Optimization Approach for Analog Circuit Synthesis via One-Dimensional Subspaces [11.64233949999656]
  We propose a fast and robust Bayesian optimization approach via one-dimensional subspaces for analog circuit synthesis. Our proposed algorithm can accelerate the optimization procedure by up to 9x and 38x compared to LP-EI and REMBOpBO respectively when the batch size is 15.
  arXiv  Detail & Related papers  (2021-09-01T21:25:25Z)
• BOSH: Bayesian Optimization by Sampling Hierarchically [10.10241176664951]
  We propose a novel BO routine pairing a hierarchical Gaussian process with an information-theoretic framework to generate a growing pool of realizations. We demonstrate that BOSH provides more efficient and higher-precision optimization than standard BO across synthetic benchmarks, simulation optimization, reinforcement learning and hyper- parameter tuning tasks.
  arXiv  Detail & Related papers  (2020-07-02T07:35:49Z)
• Incorporating Expert Prior in Bayesian Optimisation via Space Warping [54.412024556499254]
  In big search spaces the algorithm goes through several low function value regions before reaching the optimum of the function. One approach to subside this cold start phase is to use prior knowledge that can accelerate the optimisation. In this paper, we represent the prior knowledge about the function optimum through a prior distribution. The prior distribution is then used to warp the search space in such a way that space gets expanded around the high probability region of function optimum and shrinks around low probability region of optimum.
  arXiv  Detail & Related papers  (2020-03-27T06:18:49Z)
• Preferential Batch Bayesian Optimization [16.141259199997005]
  preferential batch Bayesian optimization (PBBO) is a new framework that allows finding the optimum of a latent function of interest. We show how the acquisitions developed under this framework generalize and augment previous approaches in Bayesian optimization.
  arXiv  Detail & Related papers  (2020-03-25T14:59:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.