Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Data

• URL: http://arxiv.org/abs/2405.14492v3
• Date: Tue, 03 Jun 2025 15:38:15 GMT
• Title: Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Data
• Authors: Tim Gyger, Reinhard Furrer, Fabio Sigrist,
• Abstract summary: We show how iterative methods can be used to reduce computational costs in calculating likelihoods, gradients, and predictive distributions with full-scale approximations (FSAs)<n>We introduce a novel preconditioner and show theoretically and empirically that it accelerates the conjugate gradient method's convergence speed and mitigates its sensitivity with respect to the FSA parameters.<n>In our experiments, it outperforms existing state-of-the-art preconditioners for Vecchia approximations.
• Score: 9.913418444556486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaussian processes are flexible probabilistic regression models which are widely used in statistics and machine learning. However, a drawback is their limited scalability to large data sets. To alleviate this, full-scale approximations (FSAs) combine predictive process methods and covariance tapering, thus approximating both global and local structures. We show how iterative methods can be used to reduce computational costs in calculating likelihoods, gradients, and predictive distributions with FSAs. In particular, we introduce a novel preconditioner and show theoretically and empirically that it accelerates the conjugate gradient method's convergence speed and mitigates its sensitivity with respect to the FSA parameters and the eigenvalue structure of the original covariance matrix, and we demonstrate empirically that it outperforms a state-of-the-art pivoted Cholesky preconditioner. Furthermore, we introduce an accurate and fast way to calculate predictive variances using stochastic simulation and iterative methods. In addition, we show how our newly proposed FITC preconditioner can also be used in iterative methods for Vecchia approximations. In our experiments, it outperforms existing state-of-the-art preconditioners for Vecchia approximations. All methods are implemented in a free C++ software library with high-level Python and R packages.

Related papers

• Vecchia-Inducing-Points Full-Scale Approximations for Gaussian Processes [9.913418444556486]
  We propose Vecchia-inducing-points full-scale (VIF) approximations for Gaussian processes.<n>We show that VIF approximations are both computationally efficient as well as more accurate and numerically stable than state-of-the-art alternatives.<n>All methods are implemented in the open source C++ library GPBoost with high-level Python and R interfaces.
  arXiv  Detail & Related papers  (2025-07-07T14:49:06Z)
• Scalable Computations for Generalized Mixed Effects Models with Crossed Random Effects Using Krylov Subspace Methods [11.141688859736805]
  We present novel Krylov subspace-based methods that address several existing computational bottlenecks.<n>Our software implementation is up to 10'000 times faster and more stable than state-of-the-art implementations such as lme4 and glmmTMB.
  arXiv  Detail & Related papers  (2025-05-14T16:50:19Z)
• An accuracy-runtime trade-off comparison of scalable Gaussian process approximations for spatial data [11.141688859736805]
  A drawback of Gaussian processes is their computational cost having $mathcalO(N3)$ time and $mathcalO(N2)$ memory complexity. Numerous approximation techniques have been proposed to address this limitation. We analyze this trade-off between accuracy and runtime on multiple simulated and large-scale real-world datasets.
  arXiv  Detail & Related papers  (2025-01-20T12:35:58Z)
• Linear cost and exponentially convergent approximation of Gaussian Matérn processes [43.341057405337295]
  computational cost for inference and prediction of statistical models based on Gaussian processes scales cubicly with the number of observations. We develop a method with linear cost and with a covariance error that decreases exponentially fast in the order $m$ of the proposed approximation. The method is based on an optimal rational approximation of the spectral density and results in an approximation that can be represented as a sum of $m$ independent Markov processes.
  arXiv  Detail & Related papers  (2024-10-16T19:57:15Z)
• Iterative Methods for Vecchia-Laplace Approximations for Latent Gaussian Process Models [11.141688859736805]
  We introduce and analyze several preconditioners, derive new convergence results, and propose novel methods for accurately approxing predictive variances. In particular, we obtain a speed-up of an order of magnitude compared to Cholesky-based calculations. All methods are implemented in a free C++ software library with high-level Python and R packages.
  arXiv  Detail & Related papers  (2023-10-18T14:31:16Z)
• Likelihood-based inference and forecasting for trawl processes: a stochastic optimization approach [0.0]
  We develop the first likelihood-based methodology for the inference of real-valued trawl processes. We introduce novel deterministic and probabilistic forecasting methods. We release a Python library which can be used to fit a large class of trawl processes.
  arXiv  Detail & Related papers  (2023-08-30T15:37:48Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• Posterior and Computational Uncertainty in Gaussian Processes [52.26904059556759]
  Gaussian processes scale prohibitively with the size of the dataset. Many approximation methods have been developed, which inevitably introduce approximation error. This additional source of uncertainty, due to limited computation, is entirely ignored when using the approximate posterior. We develop a new class of methods that provides consistent estimation of the combined uncertainty arising from both the finite number of data observed and the finite amount of computation expended.
  arXiv  Detail & Related papers  (2022-05-30T22:16:25Z)
• Improved Convergence Rates for Sparse Approximation Methods in Kernel-Based Learning [48.08663378234329]
  Kernel-based models such as kernel ridge regression and Gaussian processes are ubiquitous in machine learning applications. Existing sparse approximation methods can yield a significant reduction in the computational cost. We provide novel confidence intervals for the Nystr"om method and the sparse variational Gaussian processes approximation method.
  arXiv  Detail & Related papers  (2022-02-08T17:22:09Z)
• Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition [54.07797071198249]
  We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability. We demonstrate that, on a range of regression and classification problems, our approach can exploit input space symmetries such as translations and reflections. Notably, our approach achieves state-of-the-art results on CIFAR-10 among pure GP models.
  arXiv  Detail & Related papers  (2021-06-10T18:17:57Z)
• Real-Time Regression with Dividing Local Gaussian Processes [62.01822866877782]
  Local Gaussian processes are a novel, computationally efficient modeling approach based on Gaussian process regression. Due to an iterative, data-driven division of the input space, they achieve a sublinear computational complexity in the total number of training points in practice. A numerical evaluation on real-world data sets shows their advantages over other state-of-the-art methods in terms of accuracy as well as prediction and update speed.
  arXiv  Detail & Related papers  (2020-06-16T18:43:31Z)
• Instability, Computational Efficiency and Statistical Accuracy [101.32305022521024]
  We develop a framework that yields statistical accuracy based on interplay between the deterministic convergence rate of the algorithm at the population level, and its degree of (instability) when applied to an empirical object based on $n$ samples. We provide applications of our general results to several concrete classes of models, including Gaussian mixture estimation, non-linear regression models, and informative non-response models.
  arXiv  Detail & Related papers  (2020-05-22T22:30:52Z)
• Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties [12.068153197381575]
  We propose a novel variational family that allows for retaining covariances between latent processes while achieving fast convergence. We provide an efficient implementation of our new approach and apply it to several benchmark datasets. It yields excellent results and strikes a better balance between accuracy and calibrated uncertainty estimates than its state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2020-05-22T11:10:59Z)
• Global Optimization of Gaussian processes [52.77024349608834]
  We propose a reduced-space formulation with trained Gaussian processes trained on few data points. The approach also leads to significantly smaller and computationally cheaper sub solver for lower bounding. In total, we reduce time convergence by orders of orders of the proposed method.
  arXiv  Detail & Related papers  (2020-05-21T20:59:11Z)
• SLEIPNIR: Deterministic and Provably Accurate Feature Expansion for Gaussian Process Regression with Derivatives [86.01677297601624]
  We propose a novel approach for scaling GP regression with derivatives based on quadrature Fourier features. We prove deterministic, non-asymptotic and exponentially fast decaying error bounds which apply for both the approximated kernel as well as the approximated posterior.
  arXiv  Detail & Related papers  (2020-03-05T14:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.