An Empirical Study on Ensemble-Based Transfer Learning Bayesian Optimisation with Mixed Variable Types

• URL: http://arxiv.org/abs/2601.15640v1
• Date: Thu, 22 Jan 2026 04:41:26 GMT
• Title: An Empirical Study on Ensemble-Based Transfer Learning Bayesian Optimisation with Mixed Variable Types
• Authors: Natasha Trinkle, Huong Ha, Jeffrey Chan,
• Abstract summary: Historic datasets from related problems can be exploited to help improve performance of Bayesian optimisation.<n>We perform an empirical analysis of various ensemble-based transfer learning Bayesian optimisation methods and pipeline components.<n>We find that in general, two components that help improve transfer learning Bayesian optimisation performance are warm start initialisation and constraining weights used with ensemble surrogate model to be positive.
• Score: 5.417059608633818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bayesian optimisation is a sample efficient method for finding a global optimum of expensive black-box objective functions. Historic datasets from related problems can be exploited to help improve performance of Bayesian optimisation by adapting transfer learning methods to various components of the Bayesian optimisation pipeline. In this study we perform an empirical analysis of various ensemble-based transfer learning Bayesian optimisation methods and pipeline components. We expand on previous work in the literature by contributing some specific pipeline components, and three new real-time transfer learning Bayesian optimisation benchmarks. In particular we propose to use a weighting strategy for ensemble surrogate model predictions based on regularised regression with weights constrained to be positive, and a related component for handling the case when transfer learning is not improving Bayesian optimisation performance. We find that in general, two components that help improve transfer learning Bayesian optimisation performance are warm start initialisation and constraining weights used with ensemble surrogate model to be positive.

Related papers

• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Enhanced Bayesian Optimization via Preferential Modeling of Abstract Properties [49.351577714596544]
  We propose a human-AI collaborative Bayesian framework to incorporate expert preferences about unmeasured abstract properties into surrogate modeling. We provide an efficient strategy that can also handle any incorrect/misleading expert bias in preferential judgments.
  arXiv  Detail & Related papers  (2024-02-27T09:23:13Z)
• Data-driven Prior Learning for Bayesian Optimisation [5.199765487172328]
  We show that PLeBO and prior transfer find good inputs in fewer evaluations. We validate the learned priors and compare to a breadth of transfer learning approaches. We show that PLeBO and prior transfer find good inputs in fewer evaluations.
  arXiv  Detail & Related papers  (2023-11-24T18:37:52Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)
• Self-Correcting Bayesian Optimization through Bayesian Active Learning [46.235017111395344]
  We present two acquisition functions that explicitly prioritize hyperparameter learning. We then introduce Self-Correcting Bayesian Optimization (SCoreBO), which extends SAL to perform Bayesian optimization and active learning simultaneously.
  arXiv  Detail & Related papers  (2023-04-21T14:50:53Z)
• Optimizer Amalgamation [124.33523126363728]
  We are motivated to study a new problem named Amalgamation: how can we best combine a pool of "teacher" amalgamations into a single "student" that can have stronger problem-specific performance? First, we define three differentiable mechanisms to amalgamate a pool of analyticals by gradient descent. In order to reduce variance of the process, we also explore methods to stabilize the process by perturbing the target.
  arXiv  Detail & Related papers  (2022-03-12T16:07:57Z)
• Transfer Learning with Gaussian Processes for Bayesian Optimization [9.933956770453438]
  We provide a unified view on hierarchical GP models for transfer learning, which allows us to analyze the relationship between methods. We develop a novel closed-form boosted GP transfer model that fits between existing approaches in terms of complexity. We evaluate the performance of the different approaches in large-scale experiments and highlight strengths and weaknesses of the different transfer-learning methods.
  arXiv  Detail & Related papers  (2021-11-22T14:09:45Z)
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation? [66.39551991177542]
  This paper presents a sample methodology for global optimisation. Within this, a crucial performance-determiningtrivial is maximising the acquisition function. We highlight the empirical advantages of the approach to optimise functionation across 3958 individual experiments.
  arXiv  Detail & Related papers  (2020-12-15T12:18:38Z)
• Bayesian task embedding for few-shot Bayesian optimization [2.722899166098863]
  We describe a method for Bayesian optimization by which one may incorporate data from multiple systems whose quantitative ins are unknown a priori. All general (nonreal-valued) features of the systems are associated as associated inputs into a single metamodel that simultaneously learns the response surfaces of systems. We explain how the resulting probabilistic metamodel may be used for Bayesian optimization tasks and demonstrate its implementation on a variety of synthetic and real-world examples.
  arXiv  Detail & Related papers  (2020-01-02T21:46:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.