FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch

• URL: http://arxiv.org/abs/2406.04824v2
• Date: Mon, 1 Jul 2024 04:48:24 GMT
• Title: FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch
• Authors: Virginia Aglietti, Ira Ktena, Jessica Schrouff, Eleni Sgouritsa, Francisco J. R. Ruiz, Alan Malek, Alexis Bellot, Silvia Chiappa,
• Abstract summary: We show how FunBO can be used to learn new acquisition functions written in computer code. We show how FunBO identifies AFs that generalize well in and out of the training distribution of functions.
• Score: 21.41322548859776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AF can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a limited number of evaluations for a set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.

Related papers

• Discovering Preference Optimization Algorithms with and for Large Language Models [50.843710797024805]
  offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. We perform objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Experiments demonstrate the state-of-the-art performance of DiscoPOP, a novel algorithm that adaptively blends logistic and exponential losses.
  arXiv  Detail & Related papers  (2024-06-12T16:58:41Z)
• Reinforced In-Context Black-Box Optimization [64.25546325063272]
  RIBBO is a method to reinforce-learn a BBO algorithm from offline data in an end-to-end fashion. RIBBO employs expressive sequence models to learn the optimization histories produced by multiple behavior algorithms and tasks. Central to our method is to augment the optimization histories with textitregret-to-go tokens, which are designed to represent the performance of an algorithm based on cumulative regret over the future part of the histories.
  arXiv  Detail & Related papers  (2024-02-27T11:32:14Z)
• On the development of a Bayesian optimisation framework for complex unknown systems [11.066706766632578]
  This paper studies and compares common Bayesian optimisation algorithms empirically on a range of synthetic test functions. It investigates the choice of acquisition function and number of training samples, exact calculation of acquisition functions and Monte Carlo based approaches.
  arXiv  Detail & Related papers  (2022-07-19T09:50:34Z)
• Pre-training helps Bayesian optimization too [49.28382118032923]
  We seek an alternative practice for setting functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. Our results show that our method is able to locate good hyper parameters at least 3 times more efficiently than the best competing methods.
  arXiv  Detail & Related papers  (2022-07-07T04:42:54Z)
• Dynamic Multi-objective Ensemble of Acquisition Functions in Batch Bayesian Optimization [1.1602089225841632]
  The acquisition function plays a crucial role in the optimization process. Three acquisition functions are dynamically selected from a set based on their current and historical performance. Using an evolutionary multi-objective algorithm to optimize such a MOP, a set of non-dominated solutions can be obtained.
  arXiv  Detail & Related papers  (2022-06-22T14:09:18Z)
• Efficient Neural Network Analysis with Sum-of-Infeasibilities [64.31536828511021]
  Inspired by sum-of-infeasibilities methods in convex optimization, we propose a novel procedure for analyzing verification queries on networks with extensive branching functions. An extension to a canonical case-analysis-based complete search procedure can be achieved by replacing the convex procedure executed at each search state with DeepSoI.
  arXiv  Detail & Related papers  (2022-03-19T15:05:09Z)
• Optimizing Bayesian acquisition functions in Gaussian Processes [0.0]
  This paper analyzes different acquistion functions like Probability of Maximum Improvement and Expected Improvement. Along with the analysis of time taken, the paper also shows the importance of position of initial samples chosen.
  arXiv  Detail & Related papers  (2021-11-09T03:25:15Z)
• Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information [78.78486761923855]
  In many real world problems, we want to infer some property of an expensive black-box function f, given a budget of T function evaluations. We present a procedure, InfoBAX, that sequentially chooses queries that maximize mutual information with respect to the algorithm's output. On these problems, InfoBAX uses up to 500 times fewer queries to f than required by the original algorithm.
  arXiv  Detail & Related papers  (2021-04-19T17:22:11Z)
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation? [66.39551991177542]
  This paper presents a sample methodology for global optimisation. Within this, a crucial performance-determiningtrivial is maximising the acquisition function. We highlight the empirical advantages of the approach to optimise functionation across 3958 individual experiments.
  arXiv  Detail & Related papers  (2020-12-15T12:18:38Z)
• BOP-Elites, a Bayesian Optimisation algorithm for Quality-Diversity search [0.0]
  We propose the Bayesian optimisation of Elites (BOP-Elites) algorithm. By considering user defined regions of the feature space as 'niches' our task is to find the optimal solution in each niche. The resulting algorithm is very effective in identifying the parts of the search space that belong to a niche in feature space, and finding the optimal solution in each niche.
  arXiv  Detail & Related papers  (2020-05-08T23:49:13Z)
• Composition of kernel and acquisition functions for High Dimensional Bayesian Optimization [0.1749935196721634]
  We use the addition-ality of the objective function into mapping both the kernel and the acquisition function of the Bayesian Optimization. This ap-proach makes more efficient the learning/updating of the probabilistic surrogate model. Results are presented for real-life application, that is the control of pumps in urban water distribution systems.
  arXiv  Detail & Related papers  (2020-03-09T15:45:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.