On the Gaussian process limit of Bayesian Additive Regression Trees

• URL: http://arxiv.org/abs/2410.20289v1
• Date: Sat, 26 Oct 2024 23:18:33 GMT
• Title: On the Gaussian process limit of Bayesian Additive Regression Trees
• Authors: Giacomo Petrillo,
• Abstract summary: Bayesian Additive Regression Trees (BART) is a nonparametric Bayesian regression technique of rising fame. In the limit of infinite trees, it becomes equivalent to Gaussian process (GP) regression. This study opens new ways to understand and develop BART and GP regression.
• Score: 0.0
• License:
• Abstract: Bayesian Additive Regression Trees (BART) is a nonparametric Bayesian regression technique of rising fame. It is a sum-of-decision-trees model, and is in some sense the Bayesian version of boosting. In the limit of infinite trees, it becomes equivalent to Gaussian process (GP) regression. This limit is known but has not yet led to any useful analysis or application. For the first time, I derive and compute the exact BART prior covariance function. With it I implement the infinite trees limit of BART as GP regression. Through empirical tests, I show that this limit is worse than standard BART in a fixed configuration, but also that tuning the hyperparameters in the natural GP way yields a competitive method, although a properly tuned BART is still superior. The advantage of using a GP surrogate of BART is the analytical likelihood, which simplifies model building and sidesteps the complex BART MCMC. More generally, this study opens new ways to understand and develop BART and GP regression. The implementation of BART as GP is available in the Python package https://github.com/Gattocrucco/lsqfitgp .

Related papers

• Very fast Bayesian Additive Regression Trees on GPU [0.0]
  I present a GPU-enabled implementation of BART, faster by up to 200x relative to a single CPU core, making BART competitive in running time with XGBoost. This implementation is available in the Python package bartz.
  arXiv  Detail & Related papers  (2024-10-30T17:29:03Z)
• ASBART:Accelerated Soft Bayes Additive Regression Trees [8.476756500467689]
  Soft BART improves both practically and heoretically on existing Bayesian sum-of-trees models. Compared to BART,it use more than about 20 times to complete the calculation with the default setting. We proposed a variant of BART named accelerate Soft BART(ASBART)
  arXiv  Detail & Related papers  (2023-10-21T11:27:42Z)
• Revisiting Weighted Strategy for Non-stationary Parametric Bandits [82.1942459195896]
  This paper revisits the weighted strategy for non-stationary parametric bandits. We propose a refined analysis framework, which produces a simpler weight-based algorithm. Our new framework can be used to improve regret bounds of other parametric bandits.
  arXiv  Detail & Related papers  (2023-03-05T15:11:14Z)
• SoftBart: Soft Bayesian Additive Regression Trees [2.969705152497174]
  This paper introduces the SoftBart package for fitting the Soft BART algorithm of Linero and Yang. A major goal of this package has been to facilitate the inclusion of BART in larger models. I show both how to use this package for standard prediction tasks and how to embed BART models in larger models.
  arXiv  Detail & Related papers  (2022-10-28T19:25:45Z)
• GP-BART: a novel Bayesian additive regression trees approach using Gaussian processes [1.03590082373586]
  The GP-BART model is an extension of BART which addresses the limitation by assuming GP priors for the predictions of each terminal node among all trees. The model's effectiveness is demonstrated through applications to simulated and real-world data, surpassing the performance of traditional modeling approaches in various scenarios.
  arXiv  Detail & Related papers  (2022-04-05T11:18:44Z)
• Scaling Gaussian Process Optimization by Evaluating a Few Unique Candidates Multiple Times [119.41129787351092]
  We show that sequential black-box optimization based on GPs can be made efficient by sticking to a candidate solution for multiple evaluation steps. We modify two well-established GP-Opt algorithms, GP-UCB and GP-EI to adapt rules from batched GP-Opt.
  arXiv  Detail & Related papers  (2022-01-30T20:42:14Z)
• Online nonparametric regression with Sobolev kernels [99.12817345416846]
  We derive the regret upper bounds on the classes of Sobolev spaces $W_pbeta(mathcalX)$, $pgeq 2, beta>fracdp$. The upper bounds are supported by the minimax regret analysis, which reveals that in the cases $beta> fracd2$ or $p=infty$ these rates are (essentially) optimal.
  arXiv  Detail & Related papers  (2021-02-06T15:05:14Z)
• BRST-BV Quantum Actions for Constrained Totally-Symmetric Integer HS Fields [77.34726150561087]
  A non-minimal constrained BRST-BV Lagrangian formulation is presented for totally symmetric massless HS fields in a $d$-dimensional Minkowski space. A Fock-space quantum action is constructed as a shift of the total generalized field-antifield vector by a variational derivative of the gauge-fixing Fermion $Psi$ in a total BRST-BV action.
  arXiv  Detail & Related papers  (2020-10-29T16:40:32Z)
• Bayesian Additive Regression Trees with Model Trees [0.0]
  We introduce an extension of BART, called Model Trees BART (MOTR-BART) MOTR-BART considers piecewise linear functions at node levels instead of piecewise constants. In our approach, local linearities are captured more efficiently and fewer trees are required to achieve equal or better performance than BART.
  arXiv  Detail & Related papers  (2020-06-12T22:19:58Z)
• Learning Near Optimal Policies with Low Inherent Bellman Error [115.16037976819331]
  We study the exploration problem with approximate linear action-value functions in episodic reinforcement learning. We show that exploration is possible using only emphbatch assumptions with an algorithm that achieves the optimal statistical rate for the setting we consider.
  arXiv  Detail & Related papers  (2020-02-29T02:02:40Z)
• Near-linear Time Gaussian Process Optimization with Adaptive Batching and Resparsification [119.41129787351092]
  We introduce BBKB, the first no-regret GP optimization algorithm that provably runs in near-linear time and selects candidates in batches. We show that the same bound can be used to adaptively delay costly updates to the sparse GP approximation, achieving a near-constant per-step amortized cost.
  arXiv  Detail & Related papers  (2020-02-23T17:43:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.