A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimization

• URL: http://arxiv.org/abs/2501.18756v1
• Date: Thu, 30 Jan 2025 21:15:00 GMT
• Title: A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimization
• Authors: Nuojin Cheng, Leonard Papenmeier, Stephen Becker, Luigi Nardi,
• Abstract summary: We introduce a unified theoretical framework, Variational Entropy Search, which reveals that EI and information-theoretic acquisition functions are more closely related than previously recognized. We propose VES-Gamma, a novel acquisition function that balances the strengths of EI and MES.
• Score: 6.745285924097932
• License:
• Abstract: Bayesian optimization is a widely used method for optimizing expensive black-box functions, with Expected Improvement being one of the most commonly used acquisition functions. In contrast, information-theoretic acquisition functions aim to reduce uncertainty about the function's optimum and are often considered fundamentally distinct from EI. In this work, we challenge this prevailing perspective by introducing a unified theoretical framework, Variational Entropy Search, which reveals that EI and information-theoretic acquisition functions are more closely related than previously recognized. We demonstrate that EI can be interpreted as a variational inference approximation of the popular information-theoretic acquisition function, named Max-value Entropy Search. Building on this insight, we propose VES-Gamma, a novel acquisition function that balances the strengths of EI and MES. Extensive empirical evaluations across both low- and high-dimensional synthetic and real-world benchmarks demonstrate that VES-Gamma is competitive with state-of-the-art acquisition functions and in many cases outperforms EI and MES.

Related papers

• Variational Entropy Search for Adjusting Expected Improvement [3.04585143845864]
  Expected Improvement (EI) is the most commonly utilized acquisition function in black-box functions. We have developed the Variational Entropy Search (VES) methodology and the VES-Gamma algorithm, which adapts EI by incorporating principles from information-theoretic concepts.
  arXiv  Detail & Related papers  (2024-02-17T17:37:53Z)
• Unexpected Improvements to Expected Improvement for Bayesian Optimization [21.901803477674264]
  We propose LogEI, a new family of acquisition functions whose members either have identical or approximately equal optima as their canonical counterparts, but are substantially easier to optimize numerically. Our empirical results show that members of the LogEI family of acquisition functions substantially improve on the optimization performance of their canonical counterparts and surprisingly, are on par with or exceed the performance of recent state-of-the-art acquisition functions.
  arXiv  Detail & Related papers  (2023-10-31T17:59:56Z)
• Federated Conditional Stochastic Optimization [110.513884892319]
  Conditional optimization has found in a wide range of machine learning tasks, such as in-variant learning tasks, AUPRC, andAML. This paper proposes algorithms for distributed federated learning.
  arXiv  Detail & Related papers  (2023-10-04T01:47:37Z)
• Generalizing Bayesian Optimization with Decision-theoretic Entropies [102.82152945324381]
  We consider a generalization of Shannon entropy from work in statistical decision theory. We first show that special cases of this entropy lead to popular acquisition functions used in BO procedures. We then show how alternative choices for the loss yield a flexible family of acquisition functions.
  arXiv  Detail & Related papers  (2022-10-04T04:43:58Z)
• Joint Entropy Search For Maximally-Informed Bayesian Optimization [38.10887297038352]
  We propose a novel information-theoretic acquisition function that considers the entropy over the joint optimal probability density over both input and output space. Joint Entropy Search (JES) shows superior decision-making, and yields state-of-the-art performance for information-theoretic approaches across a wide suite of tasks.
  arXiv  Detail & Related papers  (2022-06-09T21:19:07Z)
• Surrogate modeling for Bayesian optimization beyond a single Gaussian process [62.294228304646516]
  We propose a novel Bayesian surrogate model to balance exploration with exploitation of the search space. To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model. To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret.
  arXiv  Detail & Related papers  (2022-05-27T16:43:10Z)
• Rectified Max-Value Entropy Search for Bayesian Optimization [54.26984662139516]
  We develop a rectified MES acquisition function based on the notion of mutual information. As a result, RMES shows a consistent improvement over MES in several synthetic function benchmarks and real-world optimization problems.
  arXiv  Detail & Related papers  (2022-02-28T08:11:02Z)
• Trusted-Maximizers Entropy Search for Efficient Bayesian Optimization [39.824086260578646]
  This paper presents a novel trusted-maximizers entropy search (TES) acquisition function. It measures how much an input contributes to the information gain on a query over a finite set of trusted maximizers.
  arXiv  Detail & Related papers  (2021-07-30T07:25:07Z)
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation? [66.39551991177542]
  This paper presents a sample methodology for global optimisation. Within this, a crucial performance-determiningtrivial is maximising the acquisition function. We highlight the empirical advantages of the approach to optimise functionation across 3958 individual experiments.
  arXiv  Detail & Related papers  (2020-12-15T12:18:38Z)
• Bilevel Optimization: Convergence Analysis and Enhanced Design [63.64636047748605]
  Bilevel optimization is a tool for many machine learning problems. We propose a novel stoc-efficientgradient estimator named stoc-BiO.
  arXiv  Detail & Related papers  (2020-10-15T18:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.