Vecchia-Inducing-Points Full-Scale Approximations for Gaussian Processes

• URL: http://arxiv.org/abs/2507.05064v1
• Date: Mon, 07 Jul 2025 14:49:06 GMT
• Title: Vecchia-Inducing-Points Full-Scale Approximations for Gaussian Processes
• Authors: Tim Gyger, Reinhard Furrer, Fabio Sigrist,
• Abstract summary: We propose Vecchia-inducing-points full-scale (VIF) approximations for Gaussian processes.<n>We show that VIF approximations are both computationally efficient as well as more accurate and numerically stable than state-of-the-art alternatives.<n>All methods are implemented in the open source C++ library GPBoost with high-level Python and R interfaces.
• Score: 9.913418444556486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaussian processes are flexible, probabilistic, non-parametric models widely used in machine learning and statistics. However, their scalability to large data sets is limited by computational constraints. To overcome these challenges, we propose Vecchia-inducing-points full-scale (VIF) approximations combining the strengths of global inducing points and local Vecchia approximations. Vecchia approximations excel in settings with low-dimensional inputs and moderately smooth covariance functions, while inducing point methods are better suited to high-dimensional inputs and smoother covariance functions. Our VIF approach bridges these two regimes by using an efficient correlation-based neighbor-finding strategy for the Vecchia approximation of the residual process, implemented via a modified cover tree algorithm. We further extend our framework to non-Gaussian likelihoods by introducing iterative methods that substantially reduce computational costs for training and prediction by several orders of magnitudes compared to Cholesky-based computations when using a Laplace approximation. In particular, we propose and compare novel preconditioners and provide theoretical convergence results. Extensive numerical experiments on simulated and real-world data sets show that VIF approximations are both computationally efficient as well as more accurate and numerically stable than state-of-the-art alternatives. All methods are implemented in the open source C++ library GPBoost with high-level Python and R interfaces.

Related papers

• Scalable Computations for Generalized Mixed Effects Models with Crossed Random Effects Using Krylov Subspace Methods [11.141688859736805]
  We present novel Krylov subspace-based methods that address several existing computational bottlenecks.<n>Our software implementation is up to 10'000 times faster and more stable than state-of-the-art implementations such as lme4 and glmmTMB.
  arXiv  Detail & Related papers  (2025-05-14T16:50:19Z)
• Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Data [9.913418444556486]
  We show how iterative methods can be used to reduce computational costs in calculating likelihoods, gradients, and predictive distributions with full-scale approximations (FSAs)<n>We introduce a novel preconditioner and show theoretically and empirically that it accelerates the conjugate gradient method's convergence speed and mitigates its sensitivity with respect to the FSA parameters.<n>In our experiments, it outperforms existing state-of-the-art preconditioners for Vecchia approximations.
  arXiv  Detail & Related papers  (2024-05-23T12:25:22Z)
• Differentially Private Optimization with Sparse Gradients [60.853074897282625]
  We study differentially private (DP) optimization problems under sparsity of individual gradients. Building on this, we obtain pure- and approximate-DP algorithms with almost optimal rates for convex optimization with sparse gradients.
  arXiv  Detail & Related papers  (2024-04-16T20:01:10Z)
• Iterative Methods for Vecchia-Laplace Approximations for Latent Gaussian Process Models [11.141688859736805]
  We introduce and analyze several preconditioners, derive new convergence results, and propose novel methods for accurately approxing predictive variances.<n>In particular, we obtain a speed-up of an order of magnitude compared to Cholesky-based calculations.<n>All methods are implemented in a free C++ software library with high-level Python and R packages.
  arXiv  Detail & Related papers  (2023-10-18T14:31:16Z)
• Federated Conditional Stochastic Optimization [110.513884892319]
  Conditional optimization has found in a wide range of machine learning tasks, such as in-variant learning tasks, AUPRC, andAML. This paper proposes algorithms for distributed federated learning.
  arXiv  Detail & Related papers  (2023-10-04T01:47:37Z)
• Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees [57.67528738886731]
  We study the numerical stability of scalable sparse approximations based on inducing points. For low-dimensional tasks such as geospatial modeling, we propose an automated method for computing inducing points satisfying these conditions.
  arXiv  Detail & Related papers  (2022-10-14T15:20:17Z)
• A Stochastic Bundle Method for Interpolating Networks [18.313879914379008]
  We propose a novel method for training deep neural networks that are capable of driving the empirical loss to zero. At each iteration our method constructs a maximum linear approximation, known as the bundle of the objective learning approximation.
  arXiv  Detail & Related papers  (2022-01-29T23:02:30Z)
• Probabilistic partition of unity networks: clustering based deep approximation [0.0]
  Partition of unity networks (POU-Nets) have been shown capable of realizing algebraic convergence rates for regression and solution of PDEs. We enrich POU-Nets with a Gaussian noise model to obtain a probabilistic generalization amenable to gradient-based generalizations of a maximum likelihood loss. We provide benchmarks quantifying performance in high/low-dimensions, demonstrating that convergence rates depend only on the latent dimension of data within high-dimensional space.
  arXiv  Detail & Related papers  (2021-07-07T08:02:00Z)
• Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition [54.07797071198249]
  We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability. We demonstrate that, on a range of regression and classification problems, our approach can exploit input space symmetries such as translations and reflections. Notably, our approach achieves state-of-the-art results on CIFAR-10 among pure GP models.
  arXiv  Detail & Related papers  (2021-06-10T18:17:57Z)
• Efficient Learning of Generative Models via Finite-Difference Score Matching [111.55998083406134]
  We present a generic strategy to efficiently approximate any-order directional derivative with finite difference. Our approximation only involves function evaluations, which can be executed in parallel, and no gradient computations.
  arXiv  Detail & Related papers  (2020-07-07T10:05:01Z)
• FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data [59.50904660420082]
  Federated Learning (FL) has become a popular paradigm for learning from distributed data. To effectively utilize data at different devices without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a "computation then aggregation" (CTA) model.
  arXiv  Detail & Related papers  (2020-05-22T23:07:42Z)
• Global Optimization of Gaussian processes [52.77024349608834]
  We propose a reduced-space formulation with trained Gaussian processes trained on few data points. The approach also leads to significantly smaller and computationally cheaper sub solver for lower bounding. In total, we reduce time convergence by orders of orders of the proposed method.
  arXiv  Detail & Related papers  (2020-05-21T20:59:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.