Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization

• URL: http://arxiv.org/abs/2605.20249v1
• Date: Mon, 18 May 2026 07:35:59 GMT
• Title: Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization
• Abstract summary: We introduce textbf Kernel Discovery, a framework for searching a broader kernel space beyond predefined composition rules.<n>We also propose a leave-one-out continuous ranked probability score (LOO-CRPS) as a selection criterion that penalizes overfitted kernels.<n>Our method achieves an average rank of textbf1.2 out of 17, outperforming competitive baselines.
• Score: 29.146209129804152
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gaussian Process (GP) kernels are central to Bayesian optimization (BO), yet designing effective kernels for high-dimensional problems still relies on extensive manual engineering. Existing automated approaches struggle in high dimensions for two bottlenecks: their kernel search space is limited to additions and multiplications of base kernels, and LLM-based approaches require conditioning on raw observations, which becomes infeasible due to context-length limits and the difficulty of extracting meaningful patterns. We introduce \textbf{Kernel Discovery}, a LLM-driven evolutionary framework for high-dimensional BO that searches a broader kernel space beyond predefined composition rules and does not require conditioning on observations. Motivated by the observation that directly prompting an LLM to generate kernel code yields syntactically varied but functionally identical kernels, we adopt a two-stage approach: an LLM first proposes novel mathematical forms, then a second LLM call converts each form into validated, executable code. We also propose a leave-one-out continuous ranked probability score (LOO-CRPS) as a selection criterion that penalizes overfitted kernels. On five high-dimensional BO benchmarks, our method achieves an average rank of \textbf{1.2 out of 17}, outperforming competitive baselines. We further analyze the discovered kernels to identify which kernels lead to improvements in high-dimensional BO.

Related papers

• Towards Automated Kernel Generation in the Era of LLMs [17.69471168609145]
  Kernel engineering is a time-consuming and non-scalable process.<n>Recent advances in large language models (LLMs) and agentic systems have opened new possibilities for automating kernel generation and optimization.<n>The field remains fragmented, lacking a systematic perspective for LLM-driven kernel generation.
  arXiv  Detail & Related papers  (2026-01-22T07:53:52Z)
• DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization [6.303144414273044]
  Large language models (LLMs) have unlocked many applications but also underscores the challenge of aligning them with diverse values and preferences.<n>Direct Preference Optimization (DPO) is central to alignment but constrained by fixed divergences and limited feature transformations.
  arXiv  Detail & Related papers  (2025-01-05T00:08:52Z)
• Mirror Descent on Reproducing Kernel Banach Spaces [12.716091600034543]
  This paper addresses a learning problem on Banach spaces endowed with a reproducing kernel. We propose an algorithm that employs gradient steps in the dual space of the Banach space using the reproducing kernel. To instantiate this algorithm in practice, we introduce a novel family of RKBSs with $p$-norm.
  arXiv  Detail & Related papers  (2024-11-18T02:18:32Z)
• Multiple Kernel Clustering with Dual Noise Minimization [56.009011016367744]
  Multiple kernel clustering (MKC) aims to group data by integrating complementary information from base kernels. In this paper, we rigorously define dual noise and propose a novel parameter-free MKC algorithm by minimizing them. We observe that dual noise will pollute the block diagonal structures and incur the degeneration of clustering performance, and C-noise exhibits stronger destruction than N-noise.
  arXiv  Detail & Related papers  (2022-07-13T08:37:42Z)
• Local Sample-weighted Multiple Kernel Clustering with Consensus Discriminative Graph [73.68184322526338]
  Multiple kernel clustering (MKC) is committed to achieving optimal information fusion from a set of base kernels. This paper proposes a novel local sample-weighted multiple kernel clustering model. Experimental results demonstrate that our LSWMKC possesses better local manifold representation and outperforms existing kernel or graph-based clustering algo-rithms.
  arXiv  Detail & Related papers  (2022-07-05T05:00:38Z)
• Learning "best" kernels from data in Gaussian process regression. With application to aerodynamics [0.4588028371034406]
  We introduce algorithms to select/design kernels in Gaussian process regression/kriging surrogate modeling techniques. A first class of algorithms is kernel flow, which was introduced in a context of classification in machine learning. A second class of algorithms is called spectral kernel ridge regression, and aims at selecting a "best" kernel such that the norm of the function to be approximated is minimal.
  arXiv  Detail & Related papers  (2022-06-03T07:50:54Z)
• Flow-based Kernel Prior with Application to Blind Super-Resolution [143.21527713002354]
  Kernel estimation is generally one of the key problems for blind image super-resolution (SR) This paper proposes a normalizing flow-based kernel prior (FKP) for kernel modeling. Experiments on synthetic and real-world images demonstrate that the proposed FKP can significantly improve the kernel estimation accuracy.
  arXiv  Detail & Related papers  (2021-03-29T22:37:06Z)
• End-to-end Kernel Learning via Generative Random Fourier Features [31.57596752889935]
  Random Fourier features (RFFs) provide a promising way for kernel learning in a spectral case. In this paper, we consider a one-stage process that incorporates the kernel learning and linear learner into a unifying framework.
  arXiv  Detail & Related papers  (2020-09-10T00:27:39Z)
• Kernel-Based Reinforcement Learning: A Finite-Time Analysis [53.47210316424326]
  We introduce Kernel-UCBVI, a model-based optimistic algorithm that leverages the smoothness of the MDP and a non-parametric kernel estimator of the rewards. We empirically validate our approach in continuous MDPs with sparse rewards.
  arXiv  Detail & Related papers  (2020-04-12T12:23:46Z)
• Learning Deep Kernels for Non-Parametric Two-Sample Tests [50.92621794426821]
  We propose a class of kernel-based two-sample tests, which aim to determine whether two sets of samples are drawn from the same distribution. Our tests are constructed from kernels parameterized by deep neural nets, trained to maximize test power.
  arXiv  Detail & Related papers  (2020-02-21T03:54:23Z)
• PolyScientist: Automatic Loop Transformations Combined with Microkernels for Optimization of Deep Learning Primitives [55.79741270235602]
  We develop a hybrid solution to the development of deep learning kernels. We use the advanced polyhedral technology to automatically tune the outer loops for performance.
  arXiv  Detail & Related papers  (2020-02-06T08:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.