Healing Products of Gaussian Processes
- URL: http://arxiv.org/abs/2102.07106v1
- Date: Sun, 14 Feb 2021 08:53:43 GMT
- Title: Healing Products of Gaussian Processes
- Authors: Samuel Cohen, Rendani Mbuvha, Tshilidzi Marwala, Marc Peter Deisenroth
- Abstract summary: We propose a new product-of-expert model that combines predictions of local experts by computing their Wasserstein barycenter.
In particular, we propose a new product-of-expert model that combines predictions of local experts by computing their Wasserstein barycenter.
- Score: 21.892542043785845
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Gaussian processes (GPs) are nonparametric Bayesian models that have been
applied to regression and classification problems. One of the approaches to
alleviate their cubic training cost is the use of local GP experts trained on
subsets of the data. In particular, product-of-expert models combine the
predictive distributions of local experts through a tractable product
operation. While these expert models allow for massively distributed
computation, their predictions typically suffer from erratic behaviour of the
mean or uncalibrated uncertainty quantification. By calibrating predictions via
a tempered softmax weighting, we provide a solution to these problems for
multiple product-of-expert models, including the generalised product of experts
and the robust Bayesian committee machine. Furthermore, we leverage the optimal
transport literature and propose a new product-of-expert model that combines
predictions of local experts by computing their Wasserstein barycenter, which
can be applied to both regression and classification.
Related papers
- On Least Square Estimation in Softmax Gating Mixture of Experts [78.3687645289918]
We investigate the performance of the least squares estimators (LSE) under a deterministic MoE model.
We establish a condition called strong identifiability to characterize the convergence behavior of various types of expert functions.
Our findings have important practical implications for expert selection.
arXiv Detail & Related papers (2024-02-05T12:31:18Z) - Conformal Approach To Gaussian Process Surrogate Evaluation With
Coverage Guarantees [47.22930583160043]
We propose a method for building adaptive cross-conformal prediction intervals.
The resulting conformal prediction intervals exhibit a level of adaptivity akin to Bayesian credibility sets.
The potential applicability of the method is demonstrated in the context of surrogate modeling of an expensive-to-evaluate simulator of the clogging phenomenon in steam generators of nuclear reactors.
arXiv Detail & Related papers (2024-01-15T14:45:18Z) - Mixtures of Gaussian process experts based on kernel stick-breaking
processes [0.6396288020763143]
We propose a new mixture model of Gaussian process experts based on kernel stick-breaking processes.
Our model maintains the intuitive appeal yet improve the performance of the existing models.
The model behaviour and improved predictive performance are demonstrated in experiments using six datasets.
arXiv Detail & Related papers (2023-04-26T21:23:01Z) - Sharp Calibrated Gaussian Processes [58.94710279601622]
State-of-the-art approaches for designing calibrated models rely on inflating the Gaussian process posterior variance.
We present a calibration approach that generates predictive quantiles using a computation inspired by the vanilla Gaussian process posterior variance.
Our approach is shown to yield a calibrated model under reasonable assumptions.
arXiv Detail & Related papers (2023-02-23T12:17:36Z) - MRCLens: an MRC Dataset Bias Detection Toolkit [82.44296974850639]
We introduce MRCLens, a toolkit that detects whether biases exist before users train the full model.
For the convenience of introducing the toolkit, we also provide a categorization of common biases in MRC.
arXiv Detail & Related papers (2022-07-18T21:05:39Z) - Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios.
We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
arXiv Detail & Related papers (2022-04-13T12:43:12Z) - Correlated Product of Experts for Sparse Gaussian Process Regression [2.466065249430993]
We propose a new approach based on aggregating predictions from several local and correlated experts.
Our method recovers independent Product of Experts, sparse GP and full GP in the limiting cases.
We demonstrate superior performance, in a time vs. accuracy sense, of our proposed method against state-of-the-art GP approximation methods.
arXiv Detail & Related papers (2021-12-17T14:14:08Z) - Scalable Marginal Likelihood Estimation for Model Selection in Deep
Learning [78.83598532168256]
Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties.
Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
arXiv Detail & Related papers (2021-04-11T09:50:24Z) - Gaussian Experts Selection using Graphical Models [7.530615321587948]
Local approximations reduce time complexity by dividing the original dataset into subsets and training a local expert on each subset.
We leverage techniques from the literature on undirected graphical models, using sparse precision matrices that encode conditional dependencies between experts to select the most important experts.
arXiv Detail & Related papers (2021-02-02T14:12:11Z) - Aggregating Dependent Gaussian Experts in Local Approximation [8.4159776055506]
We propose a novel approach for aggregating the Gaussian experts by detecting strong violations of conditional independence.
The dependency between experts is determined by using a Gaussian graphical model, which yields the precision matrix.
Our new method outperforms other state-of-the-art (SOTA) DGP approaches while being substantially more time-efficient than SOTA approaches.
arXiv Detail & Related papers (2020-10-17T21:49:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.