A Tutorial on Sparse Gaussian Processes and Variational Inference

• URL: http://arxiv.org/abs/2012.13962v8
• Date: Fri, 16 Apr 2021 11:21:38 GMT
• Title: A Tutorial on Sparse Gaussian Processes and Variational Inference
• Authors: Felix Leibfried, Vincent Dutordoir, ST John, Nicolas Durrande
• Abstract summary: This tutorial provides access to the basic matter for readers without prior knowledge in both GPs and VI. The framework can naturally handle a wide scope of supervised learning problems, ranging from regression with heteroscedastic and non-Gaussian likelihoods to classification problems with discrete labels, but also multilabel problems.
• Score: 12.827024991969578
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaussian processes (GPs) provide a framework for Bayesian inference that can offer principled uncertainty estimates for a large range of problems. For example, if we consider regression problems with Gaussian likelihoods, a GP model enjoys a posterior in closed form. However, identifying the posterior GP scales cubically with the number of training examples and requires to store all examples in memory. In order to overcome these obstacles, sparse GPs have been proposed that approximate the true posterior GP with pseudo-training examples. Importantly, the number of pseudo-training examples is user-defined and enables control over computational and memory complexity. In the general case, sparse GPs do not enjoy closed-form solutions and one has to resort to approximate inference. In this context, a convenient choice for approximate inference is variational inference (VI), where the problem of Bayesian inference is cast as an optimization problem -- namely, to maximize a lower bound of the log marginal likelihood. This paves the way for a powerful and versatile framework, where pseudo-training examples are treated as optimization arguments of the approximate posterior that are jointly identified together with hyperparameters of the generative model (i.e. prior and likelihood). The framework can naturally handle a wide scope of supervised learning problems, ranging from regression with heteroscedastic and non-Gaussian likelihoods to classification problems with discrete labels, but also multilabel problems. The purpose of this tutorial is to provide access to the basic matter for readers without prior knowledge in both GPs and VI. A proper exposition to the subject enables also access to more recent advances (like importance-weighted VI as well as interdomain, multioutput and deep GPs) that can serve as an inspiration for new research ideas.

Related papers

• Amortized Variational Inference for Deep Gaussian Processes [0.0]
  Deep Gaussian processes (DGPs) are multilayer generalizations of Gaussian processes (GPs) We introduce amortized variational inference for DGPs, which learns an inference function that maps each observation to variational parameters. Our method performs similarly or better than previous approaches at less computational cost.
  arXiv  Detail & Related papers  (2024-09-18T20:23:27Z)
• Shallow and Deep Nonparametric Convolutions for Gaussian Processes [0.0]
  We introduce a nonparametric process convolution formulation for GPs that alleviates weaknesses by using a functional sampling approach. We propose a composition of these nonparametric convolutions that serves as an alternative to classic deep GP models.
  arXiv  Detail & Related papers  (2022-06-17T19:03:04Z)
• Surrogate modeling for Bayesian optimization beyond a single Gaussian process [62.294228304646516]
  We propose a novel Bayesian surrogate model to balance exploration with exploitation of the search space. To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model. To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret.
  arXiv  Detail & Related papers  (2022-05-27T16:43:10Z)
• Scaling Gaussian Process Optimization by Evaluating a Few Unique Candidates Multiple Times [119.41129787351092]
  We show that sequential black-box optimization based on GPs can be made efficient by sticking to a candidate solution for multiple evaluation steps. We modify two well-established GP-Opt algorithms, GP-UCB and GP-EI to adapt rules from batched GP-Opt.
  arXiv  Detail & Related papers  (2022-01-30T20:42:14Z)
• A Variational Inference Approach to Inverse Problems with Gamma Hyperpriors [60.489902135153415]
  This paper introduces a variational iterative alternating scheme for hierarchical inverse problems with gamma hyperpriors. The proposed variational inference approach yields accurate reconstruction, provides meaningful uncertainty quantification, and is easy to implement.
  arXiv  Detail & Related papers  (2021-11-26T06:33:29Z)
• Non-Gaussian Gaussian Processes for Few-Shot Regression [71.33730039795921]
  We propose an invertible ODE-based mapping that operates on each component of the random variable vectors and shares the parameters across all of them. NGGPs outperform the competing state-of-the-art approaches on a diversified set of benchmarks and applications.
  arXiv  Detail & Related papers  (2021-10-26T10:45:25Z)
• Incremental Ensemble Gaussian Processes [53.3291389385672]
  We propose an incremental ensemble (IE-) GP framework, where an EGP meta-learner employs an it ensemble of GP learners, each having a unique kernel belonging to a prescribed kernel dictionary. With each GP expert leveraging the random feature-based approximation to perform online prediction and model update with it scalability, the EGP meta-learner capitalizes on data-adaptive weights to synthesize the per-expert predictions. The novel IE-GP is generalized to accommodate time-varying functions by modeling structured dynamics at the EGP meta-learner and within each GP learner.
  arXiv  Detail & Related papers  (2021-10-13T15:11:25Z)
• On MCMC for variationally sparse Gaussian processes: A pseudo-marginal approach [0.76146285961466]
  Gaussian processes (GPs) are frequently used in machine learning and statistics to construct powerful models. We propose a pseudo-marginal (PM) scheme that offers exact inference as well as computational gains through doubly estimators for the likelihood and large datasets.
  arXiv  Detail & Related papers  (2021-03-04T20:48:29Z)
• The FMRIB Variational Bayesian Inference Tutorial II: Stochastic Variational Bayes [1.827510863075184]
  This tutorial revisits the original FMRIB Variational Bayes tutorial. This new approach bears a lot of similarity to, and has benefited from, computational methods applied to machine learning algorithms.
  arXiv  Detail & Related papers  (2020-07-03T11:31:52Z)
• Likelihood-Free Inference with Deep Gaussian Processes [70.74203794847344]
  Surrogate models have been successfully used in likelihood-free inference to decrease the number of simulator evaluations. We propose a Deep Gaussian Process (DGP) surrogate model that can handle more irregularly behaved target distributions. Our experiments show how DGPs can outperform GPs on objective functions with multimodal distributions and maintain a comparable performance in unimodal cases.
  arXiv  Detail & Related papers  (2020-06-18T14:24:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.