Sparse within Sparse Gaussian Processes using Neighbor Information

• URL: http://arxiv.org/abs/2011.05041v3
• Date: Tue, 20 Jul 2021 05:13:14 GMT
• Title: Sparse within Sparse Gaussian Processes using Neighbor Information
• Authors: Gia-Lac Tran, Dimitrios Milios, Pietro Michiardi and Maurizio Filippone
• Abstract summary: We introduce a novel hierarchical prior, which imposes sparsity on the set of inducing variables. We experimentally show considerable computational gains compared to standard sparse GPs.
• Score: 23.48831040972227
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Approximations to Gaussian processes based on inducing variables, combined with variational inference techniques, enable state-of-the-art sparse approaches to infer GPs at scale through mini batch-based learning. In this work, we address one limitation of sparse GPs, which is due to the challenge in dealing with a large number of inducing variables without imposing a special structure on the inducing inputs. In particular, we introduce a novel hierarchical prior, which imposes sparsity on the set of inducing variables. We treat our model variationally, and we experimentally show considerable computational gains compared to standard sparse GPs when sparsity on the inducing variables is realized considering the nearest inducing inputs of a random mini-batch of the data. We perform an extensive experimental validation that demonstrates the effectiveness of our approach compared to the state-of-the-art. Our approach enables the possibility to use sparse GPs using a large number of inducing points without incurring a prohibitive computational cost.

Related papers

• Amortized Variational Inference for Deep Gaussian Processes [0.0]
  Deep Gaussian processes (DGPs) are multilayer generalizations of Gaussian processes (GPs) We introduce amortized variational inference for DGPs, which learns an inference function that maps each observation to variational parameters. Our method performs similarly or better than previous approaches at less computational cost.
  arXiv  Detail & Related papers  (2024-09-18T20:23:27Z)
• Actually Sparse Variational Gaussian Processes [20.71289963037696]
  We propose a new class of inter-domain variational GP constructed by projecting a GP onto a set of compactly supported B-spline basis functions. This allows us to very efficiently model fast-varying spatial phenomena with tens of thousands of inducing variables.
  arXiv  Detail & Related papers  (2023-04-11T09:38:58Z)
• Compound Batch Normalization for Long-tailed Image Classification [77.42829178064807]
  We propose a compound batch normalization method based on a Gaussian mixture. It can model the feature space more comprehensively and reduce the dominance of head classes. The proposed method outperforms existing methods on long-tailed image classification.
  arXiv  Detail & Related papers  (2022-12-02T07:31:39Z)
• Scaling Structured Inference with Randomization [64.18063627155128]
  We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
  arXiv  Detail & Related papers  (2021-12-07T11:26:41Z)
• Non-Gaussian Gaussian Processes for Few-Shot Regression [71.33730039795921]
  We propose an invertible ODE-based mapping that operates on each component of the random variable vectors and shares the parameters across all of them. NGGPs outperform the competing state-of-the-art approaches on a diversified set of benchmarks and applications.
  arXiv  Detail & Related papers  (2021-10-26T10:45:25Z)
• Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition [54.07797071198249]
  We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability. We demonstrate that, on a range of regression and classification problems, our approach can exploit input space symmetries such as translations and reflections. Notably, our approach achieves state-of-the-art results on CIFAR-10 among pure GP models.
  arXiv  Detail & Related papers  (2021-06-10T18:17:57Z)
• Sparse Gaussian Process Variational Autoencoders [24.86751422740643]
  Existing approaches for performing inference in GP-DGMs do not support sparse GP approximations based on points. We develop the sparse Gaussian processal variation autoencoder (GP-VAE) characterised by the use of partial inference networks for parameterising sparse GP approximations.
  arXiv  Detail & Related papers  (2020-10-20T10:19:56Z)
• Modulating Scalable Gaussian Processes for Expressive Statistical Learning [25.356503463916816]
  Gaussian process (GP) is interested in learning the statistical relationship between inputs and outputs, since it offers not only the prediction mean but also the associated variability. This article studies new scalable GP paradigms including the non-stationary heteroscedastic GP, the mixture of GPs and the latent GP, which introduce additional latent variables to modulate the outputs or inputs in order to learn richer, non-Gaussian statistical representation.
  arXiv  Detail & Related papers  (2020-08-29T06:41:45Z)
• Likelihood-Free Inference with Deep Gaussian Processes [70.74203794847344]
  Surrogate models have been successfully used in likelihood-free inference to decrease the number of simulator evaluations. We propose a Deep Gaussian Process (DGP) surrogate model that can handle more irregularly behaved target distributions. Our experiments show how DGPs can outperform GPs on objective functions with multimodal distributions and maintain a comparable performance in unimodal cases.
  arXiv  Detail & Related papers  (2020-06-18T14:24:05Z)
• Sparse Gaussian Processes Revisited: Bayesian Approaches to Inducing-Variable Approximations [27.43948386608]
  Variational inference techniques based on inducing variables provide an elegant framework for scalable estimation in Gaussian process (GP) models. In this work we challenge the common wisdom that optimizing the inducing inputs in variational framework yields optimal performance.
  arXiv  Detail & Related papers  (2020-03-06T08:53:18Z)
• SLEIPNIR: Deterministic and Provably Accurate Feature Expansion for Gaussian Process Regression with Derivatives [86.01677297601624]
  We propose a novel approach for scaling GP regression with derivatives based on quadrature Fourier features. We prove deterministic, non-asymptotic and exponentially fast decaying error bounds which apply for both the approximated kernel as well as the approximated posterior.
  arXiv  Detail & Related papers  (2020-03-05T14:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.