ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization

• URL: http://arxiv.org/abs/2605.06454v1
• Date: Thu, 07 May 2026 15:49:03 GMT
• Title: ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization
• Abstract summary: noisy acquisition estimates can lead to unstable decisions.<n>We introduce OrthoBO: a Bayesian optimization framework with ensemble surrogates and an outer log transformation.<n>We show theoretically that our estimator preserves the target, leads to variance reduction, and improves pairwise ranking stability.
• Score: 31.941089890706348
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bayesian optimization is widely used for hyperparameter optimization when model evaluations are expensive; however, noisy acquisition estimates can lead to unstable decisions. We identify acquisition estimation noise as a failure mode that was previously overlooked: even when the surrogate model and acquisition target are correctly specified, finite-sample Monte Carlo error can perturb acquisition values. This can, in turn, flip candidate rankings and lead to suboptimal BO decisions. As a remedy, we aim at variance reduction and propose an orthogonal acquisition estimator that subtracts an optimally weighted score-function control variate, which yields an acquisition residual orthogonal to posterior score directions and which thus reduces Monte Carlo variance. We further introduce OrthoBO: a Bayesian optimization framework that combines our orthogonal acquisition estimator with ensemble surrogates and an outer log transformation. We show theoretically that our estimator preserves the target, leads to variance reduction, and improves pairwise ranking stability. We further verify the theoretical properties of OrthoBO through numerical experiments where our framework reduces acquisition estimation variance, stabilizes candidate rankings, and achieves strong performance. We also demonstrate the downstream utility of OrthoBO in hyperparameter optimization for neural network training and fine-tuning.

Related papers

• Bayesian optimization with kernel ensembles and disagreement-based acquisition for source localization and acoustic inversion [12.760068476185504]
  Joint source localization and geoacoustic inversion requires optimizing an objective built from an expensive normal mode propagation model.<n>We use a weighted ensemble of GPs with different kernel families to adapt to the observed objective without committing to one kernel in advance.<n>Experiments on simulated and measured SWellEx-96 data show that the resulting method achieves the lowest mean final objective.
  arXiv  Detail & Related papers  (2026-09-13T03:52:00Z)
• On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimization [57.179679246370114]
  A potential limitation of existing methods is the bias inherent in most perturbation estimators unless a stepsize is proposed.<n>We propose a novel family of unbiased gradient scaling estimators that eliminate bias while maintaining favorable construction.
  arXiv  Detail & Related papers  (2025-10-22T18:25:43Z)
• From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models [90.45197506653341]
  Large reasoning models generate intermediate reasoning traces before producing final answers.<n> aligning LRMs with human preferences, a crucial prerequisite for model deployment, remains underexplored.<n>A common workaround optimized a single sampled trajectory, which introduces substantial gradient variance from trace sampling.
  arXiv  Detail & Related papers  (2025-10-06T17:58:01Z)
• Adaptive Bayesian Optimization for Robust Identification of Stochastic Dynamical Systems [4.0148499400442095]
  This paper deals with the identification of linear derivation systems, where the unknowns include system coefficients and noise variances.<n>A sample-efficient global optimization method based on Bayesian optimization is proposed.<n>Experiments show that the EGP-based BO consistently outperforms MLE via steady-state filtering and expectation-maximization.
  arXiv  Detail & Related papers  (2025-03-09T01:38:21Z)
• Robust Bayesian Optimization via Localized Online Conformal Prediction [37.549297668783254]
  We introduce localized online conformal prediction-based Bayesian optimization (LOCBO)<n>LOCBO calibrates the GP model through localized online conformal prediction (CP)<n>We provide theoretical performance guarantees for LOCBO's iterates that hold for the unobserved objective function.
  arXiv  Detail & Related papers  (2024-11-26T12:45:54Z)
• Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approach [51.76826149868971]
  Policy evaluation via Monte Carlo simulation is at the core of many MC Reinforcement Learning (RL) algorithms. We propose as a quality index a surrogate of the mean squared error of a return estimator that uses trajectories of different lengths. We present an adaptive algorithm called Robust and Iterative Data collection strategy Optimization (RIDO)
  arXiv  Detail & Related papers  (2024-10-17T11:47:56Z)
• Model-based Causal Bayesian Optimization [78.120734120667]
  We propose model-based causal Bayesian optimization (MCBO) MCBO learns a full system model instead of only modeling intervention-reward pairs. Unlike in standard Bayesian optimization, our acquisition function cannot be evaluated in closed form.
  arXiv  Detail & Related papers  (2022-11-18T14:28:21Z)
• Neighbor Regularized Bayesian Optimization for Hyperparameter Optimization [12.544312247050236]
  We propose a novel BO algorithm called Neighbor Regularized Bayesian Optimization (NRBO) to solve the problem. We first propose a neighbor-based regularization to smooth each sample observation, which could reduce the observation noise efficiently without any extra training cost. We conduct experiments on the bayesmark benchmark and important computer vision benchmarks such as ImageNet and COCO.
  arXiv  Detail & Related papers  (2022-10-07T12:08:01Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence [77.06203118175335]
  Variational Inference (VI) is a popular alternative to exact sampling in Bayesian inference. Importance sampling (IS) is often used to fine-tune and de-bias the estimates of approximate Bayesian inference procedures. We propose a novel combination of optimization and sampling techniques for approximate Bayesian inference.
  arXiv  Detail & Related papers  (2021-06-30T11:00:24Z)
• On Low-rank Trace Regression under General Sampling Distribution [9.699586426043885]
  We show that cross-validated estimators satisfy near-optimal error bounds on general assumptions. We also show that the cross-validated estimator outperforms the theory-inspired approach of selecting the parameter.
  arXiv  Detail & Related papers  (2019-04-18T02:56:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.