Simple and Scalable Parallelized Bayesian Optimization

• URL: http://arxiv.org/abs/2006.13600v1
• Date: Wed, 24 Jun 2020 10:25:27 GMT
• Title: Simple and Scalable Parallelized Bayesian Optimization
• Authors: Masahiro Nomura
• Abstract summary: We propose a simple and scalable BO method for asynchronous parallel settings. Experiments are carried out with a benchmark function and hyperparameter optimization of multi-layer perceptrons.
• Score: 2.512827436728378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, leveraging parallel and distributed computational resources has become essential to solve problems of high computational cost. Bayesian optimization (BO) has shown attractive results in those expensive-to-evaluate problems such as hyperparameter optimization of machine learning algorithms. While many parallel BO methods have been developed to search efficiently utilizing these computational resources, these methods assumed synchronous settings or were not scalable. In this paper, we propose a simple and scalable BO method for asynchronous parallel settings. Experiments are carried out with a benchmark function and hyperparameter optimization of multi-layer perceptrons, which demonstrate the promising performance of the proposed method.

Related papers

• Faster WIND: Accelerating Iterative Best-of-$N$ Distillation for LLM Alignment [81.84950252537618]
  This paper reveals a unified game-theoretic connection between iterative BOND and self-play alignment. We establish a novel framework, WIN rate Dominance (WIND), with a series of efficient algorithms for regularized win rate dominance optimization.
  arXiv  Detail & Related papers  (2024-10-28T04:47:39Z)
• Analyzing and Enhancing the Backward-Pass Convergence of Unrolled Optimization [50.38518771642365]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. A central challenge in this setting is backpropagation through the solution of an optimization problem, which often lacks a closed form. This paper provides theoretical insights into the backward pass of unrolled optimization, showing that it is equivalent to the solution of a linear system by a particular iterative method. A system called Folded Optimization is proposed to construct more efficient backpropagation rules from unrolled solver implementations.
  arXiv  Detail & Related papers  (2023-12-28T23:15:18Z)
• Parallel Bayesian Optimization Using Satisficing Thompson Sampling for Time-Sensitive Black-Box Optimization [0.0]
  We propose satisficing Thompson sampling-based parallel BO approaches, including synchronous and asynchronous versions. We shift the target from an optimal solution to a satisficing solution that is easier to learn. The effectiveness of the proposed methods is demonstrated on a fast-charging design problem of Lithium-ion batteries.
  arXiv  Detail & Related papers  (2023-10-19T07:03:51Z)
• An Optimization-based Deep Equilibrium Model for Hyperspectral Image Deconvolution with Convergence Guarantees [71.57324258813675]
  We propose a novel methodology for addressing the hyperspectral image deconvolution problem. A new optimization problem is formulated, leveraging a learnable regularizer in the form of a neural network. The derived iterative solver is then expressed as a fixed-point calculation problem within the Deep Equilibrium framework.
  arXiv  Detail & Related papers  (2023-06-10T08:25:16Z)
• New Paradigms for Exploiting Parallel Experiments in Bayesian Optimization [0.0]
  We present new parallel BO paradigms that exploit the structure of the system to partition the design space. Specifically, we propose an approach that partitions the design space by following the level sets of the performance function. Our results show that our approaches significantly reduce the required search time and increase the probability of finding a global (rather than local) solution.
  arXiv  Detail & Related papers  (2022-10-03T16:45:23Z)
• Outlier-Robust Sparse Estimation via Non-Convex Optimization [73.18654719887205]
  We explore the connection between high-dimensional statistics and non-robust optimization in the presence of sparsity constraints. We develop novel and simple optimization formulations for these problems. As a corollary, we obtain that any first-order method that efficiently converges to station yields an efficient algorithm for these tasks.
  arXiv  Detail & Related papers  (2021-09-23T17:38:24Z)
• Particle Swarm Optimization: Fundamental Study and its Application to Optimization and to Jetty Scheduling Problems [0.0]
  The advantages of evolutionary algorithms with respect to traditional methods have been greatly discussed in the literature. While particle swarms share such advantages, they outperform evolutionary algorithms in that they require lower computational cost and easier implementation. This paper does not intend to study their tuning, general-purpose settings are taken from previous studies, and virtually the same algorithm is used to optimize a variety of notably different problems.
  arXiv  Detail & Related papers  (2021-01-25T02:06:30Z)
• Batch Sequential Adaptive Designs for Global Optimization [5.825138898746968]
  Efficient global optimization (EGO) is one of the most popular SAD methods for expensive black-box optimization problems. For those multiple points EGO methods, the heavy computation and points clustering are the obstacles. In this work, a novel batch SAD method, named "accelerated EGO", is forwarded by using a refined sampling/importance resampling (SIR) method. The efficiency of the proposed SAD is validated by nine classic test functions with dimension from 2 to 12.
  arXiv  Detail & Related papers  (2020-10-21T01:11:35Z)
• Bilevel Optimization: Convergence Analysis and Enhanced Design [63.64636047748605]
  Bilevel optimization is a tool for many machine learning problems. We propose a novel stoc-efficientgradient estimator named stoc-BiO.
  arXiv  Detail & Related papers  (2020-10-15T18:09:48Z)
• aphBO-2GP-3B: A budgeted asynchronous parallel multi-acquisition functions for constrained Bayesian optimization on high-performing computing architecture [4.738678765150249]
  An asynchronous constrained batch-parallel Bayesian optimization method is proposed to solve the computationally-expensive simulation-based optimization problems. The advantages of this method are three-fold. The aphBO-2GP-3B framework is demonstrated using two high-fidelity expensive industrial applications.
  arXiv  Detail & Related papers  (2020-03-20T18:02:27Z)
• Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving [106.63673243937492]
  Feedforward computation, such as evaluating a neural network or sampling from an autoregressive model, is ubiquitous in machine learning. We frame the task of feedforward computation as solving a system of nonlinear equations. We then propose to find the solution using a Jacobi or Gauss-Seidel fixed-point method, as well as hybrid methods of both. Our method is guaranteed to give exactly the same values as the original feedforward computation with a reduced (or equal) number of parallelizable iterations, and hence reduced time given sufficient parallel computing power.
  arXiv  Detail & Related papers  (2020-02-10T10:11:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.