Distributional encoding for Gaussian process regression with qualitative inputs

• URL: http://arxiv.org/abs/2506.04813v1
• Date: Thu, 05 Jun 2025 09:35:02 GMT
• Title: Distributional encoding for Gaussian process regression with qualitative inputs
• Authors: Sébastien Da Veiga,
• Abstract summary: We show that a generalization based on distributional encoding (DE) makes use of all samples of the target variable for a category.<n>Our approach is validated empirically, and we demonstrate state-of-the-art predictive performance on a variety of synthetic and real-world datasets.
• Score: 0.7342677574855652
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaussian Process (GP) regression is a popular and sample-efficient approach for many engineering applications, where observations are expensive to acquire, and is also a central ingredient of Bayesian optimization (BO), a highly prevailing method for the optimization of black-box functions. However, when all or some input variables are categorical, building a predictive and computationally efficient GP remains challenging. Starting from the naive target encoding idea, where the original categorical values are replaced with the mean of the target variable for that category, we propose a generalization based on distributional encoding (DE) which makes use of all samples of the target variable for a category. To handle this type of encoding inside the GP, we build upon recent results on characteristic kernels for probability distributions, based on the maximum mean discrepancy and the Wasserstein distance. We also discuss several extensions for classification, multi-task learning and incorporation or auxiliary information. Our approach is validated empirically, and we demonstrate state-of-the-art predictive performance on a variety of synthetic and real-world datasets. DE is naturally complementary to recent advances in BO over discrete and mixed-spaces.

Related papers

• Neighbour-Driven Gaussian Process Variational Autoencoders for Scalable Structured Latent Modelling [14.358070928996069]
  Gaussian Process (GP) Variational Autoencoders (VAEs) extend standard VAEs by replacing the fully factorised Gaussian prior with a GP prior.<n> performing exact GP inference in large-scale GPVAEs is computationally prohibitive, often forcing existing approaches to rely on restrictive kernel assumptions.<n>We propose a neighbour-driven approximation strategy that exploits local adjacencies in the latent space to achieve scalable GPVAE inference.
  arXiv  Detail & Related papers  (2025-05-22T10:07:33Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Optimal Kernel Learning for Gaussian Process Models with High-Dimensional Input [0.0]
  In some simulation models, the outputs may only be significantly influenced by a small subset of the input variables, referred to as the active variables''<n>We propose an optimal kernel learning approach to identify these active variables, thereby overcoming GP model limitations and enhancing system understanding.
  arXiv  Detail & Related papers  (2025-02-23T15:39:59Z)
• Tighter sparse variational Gaussian processes [22.290236192353316]
  Sparse variational Gaussian process (GP) approximations have become the de facto standard for scaling GPs to large datasets.<n>This paper introduces a provably tighter variational approximation by relaxing the standard assumption that the conditional approximate posterior given the inducing points must match that in the prior.
  arXiv  Detail & Related papers  (2025-02-07T08:33:28Z)
• Compound Batch Normalization for Long-tailed Image Classification [77.42829178064807]
  We propose a compound batch normalization method based on a Gaussian mixture. It can model the feature space more comprehensively and reduce the dominance of head classes. The proposed method outperforms existing methods on long-tailed image classification.
  arXiv  Detail & Related papers  (2022-12-02T07:31:39Z)
• Gaussian Graphical Models as an Ensemble Method for Distributed Gaussian Processes [8.4159776055506]
  We propose a novel approach for aggregating the Gaussian experts' predictions by Gaussian graphical model (GGM) We first estimate the joint distribution of latent and observed variables using the Expectation-Maximization (EM) algorithm. Our new method outperforms other state-of-the-art DGP approaches.
  arXiv  Detail & Related papers  (2022-02-07T15:22:56Z)
• Local policy search with Bayesian optimization [73.0364959221845]
  Reinforcement learning aims to find an optimal policy by interaction with an environment. Policy gradients for local search are often obtained from random perturbations. We develop an algorithm utilizing a probabilistic model of the objective function and its gradient.
  arXiv  Detail & Related papers  (2021-06-22T16:07:02Z)
• Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition [54.07797071198249]
  We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability. We demonstrate that, on a range of regression and classification problems, our approach can exploit input space symmetries such as translations and reflections. Notably, our approach achieves state-of-the-art results on CIFAR-10 among pure GP models.
  arXiv  Detail & Related papers  (2021-06-10T18:17:57Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)
• Zeroth-Order Hybrid Gradient Descent: Towards A Principled Black-Box Optimization Framework [100.36569795440889]
  This work is on the iteration of zero-th-order (ZO) optimization which does not require first-order information. We show that with a graceful design in coordinate importance sampling, the proposed ZO optimization method is efficient both in terms of complexity as well as as function query cost.
  arXiv  Detail & Related papers  (2020-12-21T17:29:58Z)
• Sparse Gaussian Processes Revisited: Bayesian Approaches to Inducing-Variable Approximations [27.43948386608]
  Variational inference techniques based on inducing variables provide an elegant framework for scalable estimation in Gaussian process (GP) models. In this work we challenge the common wisdom that optimizing the inducing inputs in variational framework yields optimal performance.
  arXiv  Detail & Related papers  (2020-03-06T08:53:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.