Inverse Bayesian Optimization: Learning Human Search Strategies in a
Sequential Optimization Task
- URL: http://arxiv.org/abs/2104.09237v1
- Date: Fri, 16 Apr 2021 15:40:34 GMT
- Title: Inverse Bayesian Optimization: Learning Human Search Strategies in a
Sequential Optimization Task
- Authors: Nathan Sandholtz, Yohsuke Miyamoto, Luke Bornn, Maurice Smith
- Abstract summary: In this paper, we explore the inverse problem of Bayesian optimization.
We estimate the agent's latent acquisition function based on observed search paths.
We illustrate our methods by analyzing human behavior from an experiment.
- Score: 0.10499611180329801
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Bayesian optimization is a popular algorithm for sequential optimization of a
latent objective function when sampling from the objective is costly. The
search path of the algorithm is governed by the acquisition function, which
defines the agent's search strategy. Conceptually, the acquisition function
characterizes how the optimizer balances exploration and exploitation when
searching for the optimum of the latent objective. In this paper, we explore
the inverse problem of Bayesian optimization; we seek to estimate the agent's
latent acquisition function based on observed search paths. We introduce a
probabilistic solution framework for the inverse problem which provides a
principled framework to quantify both the variability with which the agent
performs the optimization task as well as the uncertainty around their
estimated acquisition function.
We illustrate our methods by analyzing human behavior from an experiment
which was designed to force subjects to balance exploration and exploitation in
search of an invisible target location. We find that while most subjects
demonstrate clear trends in their search behavior, there is significant
variation around these trends from round to round. A wide range of search
strategies are exhibited across the subjects in our study, but upper confidence
bound acquisition functions offer the best fit for the majority of subjects.
Finally, some subjects do not map well to any of the acquisition functions we
initially consider; these subjects tend to exhibit exploration preferences
beyond that of standard acquisition functions to capture. Guided by the model
discrepancies, we augment the candidate acquisition functions to yield a
superior fit to the human behavior in this task.
Related papers
- Discovering Preference Optimization Algorithms with and for Large Language Models [50.843710797024805]
offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs.
We perform objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention.
Experiments demonstrate the state-of-the-art performance of DiscoPOP, a novel algorithm that adaptively blends logistic and exponential losses.
arXiv Detail & Related papers (2024-06-12T16:58:41Z) - Mastering the exploration-exploitation trade-off in Bayesian
Optimization [0.2538209532048867]
The acquisition function drives the choice of the next solution to evaluate, balancing between exploration and exploitation.
This paper proposes a novel acquisition function, mastering the trade-off between explorative and exploitative choices, adaptively.
arXiv Detail & Related papers (2023-05-15T13:19:03Z) - DeepHive: A multi-agent reinforcement learning approach for automated
discovery of swarm-based optimization policies [0.0]
The state of each agent within the swarm is defined as its current position and function value within a design space.
The proposed approach is tested on various benchmark optimization functions and compared to the performance of other global optimization strategies.
arXiv Detail & Related papers (2023-03-29T18:08:08Z) - Generalizing Bayesian Optimization with Decision-theoretic Entropies [102.82152945324381]
We consider a generalization of Shannon entropy from work in statistical decision theory.
We first show that special cases of this entropy lead to popular acquisition functions used in BO procedures.
We then show how alternative choices for the loss yield a flexible family of acquisition functions.
arXiv Detail & Related papers (2022-10-04T04:43:58Z) - Bayesian Optimization with Informative Covariance [13.113313427848828]
We propose novel informative covariance functions for optimization, leveraging nonstationarity to encode preferences for certain regions of the search space.
We demonstrate that the proposed functions can increase the sample efficiency of Bayesian optimization in high dimensions, even under weak prior information.
arXiv Detail & Related papers (2022-08-04T15:05:11Z) - Optimizing Bayesian acquisition functions in Gaussian Processes [0.0]
This paper analyzes different acquistion functions like Probability of Maximum Improvement and Expected Improvement.
Along with the analysis of time taken, the paper also shows the importance of position of initial samples chosen.
arXiv Detail & Related papers (2021-11-09T03:25:15Z) - RoMA: Robust Model Adaptation for Offline Model-based Optimization [115.02677045518692]
We consider the problem of searching an input maximizing a black-box objective function given a static dataset of input-output queries.
A popular approach to solving this problem is maintaining a proxy model that approximates the true objective function.
Here, the main challenge is how to avoid adversarially optimized inputs during the search.
arXiv Detail & Related papers (2021-10-27T05:37:12Z) - Local policy search with Bayesian optimization [73.0364959221845]
Reinforcement learning aims to find an optimal policy by interaction with an environment.
Policy gradients for local search are often obtained from random perturbations.
We develop an algorithm utilizing a probabilistic model of the objective function and its gradient.
arXiv Detail & Related papers (2021-06-22T16:07:02Z) - Are we Forgetting about Compositional Optimisers in Bayesian
Optimisation? [66.39551991177542]
This paper presents a sample methodology for global optimisation.
Within this, a crucial performance-determiningtrivial is maximising the acquisition function.
We highlight the empirical advantages of the approach to optimise functionation across 3958 individual experiments.
arXiv Detail & Related papers (2020-12-15T12:18:38Z) - Sequential Subspace Search for Functional Bayesian Optimization
Incorporating Experimenter Intuition [63.011641517977644]
Our algorithm generates a sequence of finite-dimensional random subspaces of functional space spanned by a set of draws from the experimenter's Gaussian Process.
Standard Bayesian optimisation is applied on each subspace, and the best solution found used as a starting point (origin) for the next subspace.
We test our algorithm in simulated and real-world experiments, namely blind function matching, finding the optimal precipitation-strengthening function for an aluminium alloy, and learning rate schedule optimisation for deep networks.
arXiv Detail & Related papers (2020-09-08T06:54:11Z) - BOP-Elites, a Bayesian Optimisation algorithm for Quality-Diversity
search [0.0]
We propose the Bayesian optimisation of Elites (BOP-Elites) algorithm.
By considering user defined regions of the feature space as 'niches' our task is to find the optimal solution in each niche.
The resulting algorithm is very effective in identifying the parts of the search space that belong to a niche in feature space, and finding the optimal solution in each niche.
arXiv Detail & Related papers (2020-05-08T23:49:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.