Olympus: a benchmarking framework for noisy optimization and experiment planning

• URL: http://arxiv.org/abs/2010.04153v2
• Date: Tue, 30 Mar 2021 14:37:20 GMT
• Title: Olympus: a benchmarking framework for noisy optimization and experiment planning
• Authors: Florian H\"ase and Matteo Aldeghi and Riley J. Hickman and Lo\"ic M. Roch and Melodie Christensen and Elena Liles and Jason E. Hein and Al\'an Aspuru-Guzik
• Abstract summary: Experiment planning strategies based on off-the-shelf optimization algorithms can be employed in fully autonomous research platforms. It is unclear how their performance would translate to noisy, higher-dimensional experimental tasks. We introduce Olympus, a software package that provides a consistent and easy-to-use framework for benchmarking optimization algorithms.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Research challenges encountered across science, engineering, and economics can frequently be formulated as optimization tasks. In chemistry and materials science, recent growth in laboratory digitization and automation has sparked interest in optimization-guided autonomous discovery and closed-loop experimentation. Experiment planning strategies based on off-the-shelf optimization algorithms can be employed in fully autonomous research platforms to achieve desired experimentation goals with the minimum number of trials. However, the experiment planning strategy that is most suitable to a scientific discovery task is a priori unknown while rigorous comparisons of different strategies are highly time and resource demanding. As optimization algorithms are typically benchmarked on low-dimensional synthetic functions, it is unclear how their performance would translate to noisy, higher-dimensional experimental tasks encountered in chemistry and materials science. We introduce Olympus, a software package that provides a consistent and easy-to-use framework for benchmarking optimization algorithms against realistic experiments emulated via probabilistic deep-learning models. Olympus includes a collection of experimentally derived benchmark sets from chemistry and materials science and a suite of experiment planning strategies that can be easily accessed via a user-friendly python interface. Furthermore, Olympus facilitates the integration, testing, and sharing of custom algorithms and user-defined datasets. In brief, Olympus mitigates the barriers associated with benchmarking optimization algorithms on realistic experimental scenarios, promoting data sharing and the creation of a standard framework for evaluating the performance of experiment planning strategies

Related papers

• Enhanced Bayesian Optimization via Preferential Modeling of Abstract Properties [49.351577714596544]
  We propose a human-AI collaborative Bayesian framework to incorporate expert preferences about unmeasured abstract properties into surrogate modeling. We provide an efficient strategy that can also handle any incorrect/misleading expert bias in preferential judgments.
  arXiv  Detail & Related papers  (2024-02-27T09:23:13Z)
• Optimistic Active Exploration of Dynamical Systems [52.91573056896633]
  We develop an algorithm for active exploration called OPAX. We show how OPAX can be reduced to an optimal control problem that can be solved at each episode. Our experiments show that OPAX is not only theoretically sound but also performs well for zero-shot planning on novel downstream tasks.
  arXiv  Detail & Related papers  (2023-06-21T16:26:59Z)
• Active Exploration via Experiment Design in Markov Chains [86.41407938210193]
  A key challenge in science and engineering is to design experiments to learn about some unknown quantity of interest. We propose an algorithm that efficiently selects policies whose measurement allocation converges to the optimal one. In addition to our theoretical analysis, we showcase our framework on applications in ecological surveillance and pharmacology.
  arXiv  Detail & Related papers  (2022-06-29T00:04:40Z)
• Benchmarking Active Learning Strategies for Materials Optimization and Discovery [17.8738267360992]
  We present a reference dataset to benchmark active learning strategies in the form of various acquisition functions. We discuss the relationship between algorithm performance, materials search space, complexity, and the incorporation of prior knowledge.
  arXiv  Detail & Related papers  (2022-04-12T14:27:33Z)
• Bayesian optimization with known experimental and design constraints for chemistry applications [0.0]
  We extend our experiment planning algorithms Phoenics and Gryffin such that they can handle arbitrary known constraints. We illustrate their practical utility in two simulated chemical research scenarios.
  arXiv  Detail & Related papers  (2022-03-29T22:16:54Z)
• A Field Guide to Federated Optimization [161.3779046812383]
  Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data. This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms.
  arXiv  Detail & Related papers  (2021-07-14T18:09:08Z)
• Golem: An algorithm for robust experiment and process optimization [0.0]
  Design of experiment and optimization algorithms are often adopted to solve these tasks efficiently. Golem is an algorithm that is suboptimal to the choice of experiment planning strategy. It can be used to analyze the robustness of past experiments, or to guide experiment planning toward robust solutions on the fly.
  arXiv  Detail & Related papers  (2021-03-05T15:00:34Z)
• A User's Guide to Calibrating Robotics Simulators [54.85241102329546]
  This paper proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterize and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behavior and main properties of sim-to-real algorithms.
  arXiv  Detail & Related papers  (2020-11-17T22:24:26Z)
• Incorporating Expert Prior Knowledge into Experimental Design via Posterior Sampling [58.56638141701966]
  Experimenters can often acquire the knowledge about the location of the global optimum. It is unknown how to incorporate the expert prior knowledge about the global optimum into Bayesian optimization. An efficient Bayesian optimization approach has been proposed via posterior sampling on the posterior distribution of the global optimum.
  arXiv  Detail & Related papers  (2020-02-26T01:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.