An RKHS Perspective on Tree Ensembles

• URL: http://arxiv.org/abs/2512.00397v1
• Date: Sat, 29 Nov 2025 08:54:37 GMT
• Title: An RKHS Perspective on Tree Ensembles
• Authors: Mehdi Dagdoug, Clement Dombry, Jean-Jil Duchamps,
• Abstract summary: We develop a theoretical framework for analyzing Random Forests and Gradient Boosting.<n>We show that a Random Forest predictor can be characterized as the unique minimizer of a penalized empirical risk functional.<n>A key feature of this framework is that both the kernel and the RKHS geometry are data-dependent, offering a theoretical explanation for the strong empirical performance of tree-based ensembles.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Random Forests and Gradient Boosting are among the most effective algorithms for supervised learning on tabular data. Both belong to the class of tree-based ensemble methods, where predictions are obtained by aggregating many randomized regression trees. In this paper, we develop a theoretical framework for analyzing such methods through Reproducing Kernel Hilbert Spaces (RKHSs) constructed on tree ensembles -- more precisely, on the random partitions generated by randomized regression trees. We establish fundamental analytical properties of the resulting Random Forest kernel, including boundedness, continuity, and universality, and show that a Random Forest predictor can be characterized as the unique minimizer of a penalized empirical risk functional in this RKHS, providing a variational interpretation of ensemble learning. We further extend this perspective to the continuous-time formulation of Gradient Boosting introduced by Dombry and Duchamps, and demonstrate that it corresponds to a gradient flow on a Hilbert manifold induced by the Random Forest RKHS. A key feature of this framework is that both the kernel and the RKHS geometry are data-dependent, offering a theoretical explanation for the strong empirical performance of tree-based ensembles. Finally, we illustrate the practical potential of this approach by introducing a kernel principal component analysis built on the Random Forest kernel, which enhances the interpretability of ensemble models, as well as GVI, a new geometric variable importance criterion.

Related papers

• Learning Reconstructive Embeddings in Reproducing Kernel Hilbert Spaces via the Representer Theorem [2.0573301822495553]
  This work proposes new algorithms for reconstruction-based manifold learning within Reproducing Kernel Hilbert Spaces (RKHS)<n>A separable operator-valued kernel extends the formulation to vector-valued data while retaining the simplicity of a single scalar similarity function.<n>A subsequent kernel-alignment task projects the data into a lower-dimensional latent space whose Gram matrix aims to match the high-dimensional reconstruction kernel.
  arXiv  Detail & Related papers  (2026-01-09T14:35:19Z)
• Tree-Preconditioned Differentiable Optimization and Axioms as Layers [0.0]
  "Axioms-as-Layers" paradigm embeds axiomatic structure of Random Utility Models directly into deep neural networks.<n>"Axioms-as-Layers" paradigm eliminates the structural overfitting inherent in penalty-based methods.
  arXiv  Detail & Related papers  (2025-12-03T04:47:37Z)
• Hierarchical Bayesian Operator-induced Symbolic Regression Trees for Structural Learning of Scientific Expressions [3.8545239266455185]
  We develop a hierarchical Bayesian framework for symbolic regression that represents scientific laws as ensembles of tree-structured symbolic expressions with a regularized tree prior.<n>We establish near-minimax rate of Bayesian posterior concentration, providing the first rigorous guarantee in context of symbolic regression.<n> Empirical evaluation demonstrates robust performance of our proposed methodology against state-of-the-art competing modules.
  arXiv  Detail & Related papers  (2025-09-24T02:42:25Z)
• TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling [65.46347858249295]
  TreePO is a self-guided rollout algorithm that views sequence generation as a tree-structured searching process.<n>TreePO essentially reduces the per-update compute burden while preserving or enhancing exploration diversity.
  arXiv  Detail & Related papers  (2025-08-24T16:52:37Z)
• Learning Decision Trees as Amortized Structure Inference [59.65621207449269]
  We propose a hybrid amortized structure inference approach to learn predictive decision tree ensembles given data.<n>We show that our approach, DT-GFN, outperforms state-of-the-art decision tree and deep learning methods on standard classification benchmarks.
  arXiv  Detail & Related papers  (2025-03-10T07:05:07Z)
• PhyloGen: Language Model-Enhanced Phylogenetic Inference via Graph Structure Generation [50.80441546742053]
  Phylogenetic trees elucidate evolutionary relationships among species.<n>Traditional Markov Chain Monte Carlo methods face slow convergence and computational burdens.<n>We propose PhyloGen, a novel method leveraging a pre-trained genomic language model.
  arXiv  Detail & Related papers  (2024-12-25T08:33:05Z)
• Randomized Spline Trees for Functional Data Classification: Theory and Application to Environmental Time Series [0.0]
  This paper introduces Randomized Spline Trees (RST), a novel algorithm that bridges the two approaches by incorporating randomized functional representations into the Random Forest framework. RST generates diverse functional representations of input data using randomized B-spline parameters, creating an ensemble of decision trees trained on these varied representations. Results show that RST variants outperform standard Random Forests and Gradient Boosting on most datasets, improving classification accuracy by up to 14%.
  arXiv  Detail & Related papers  (2024-09-12T09:38:16Z)
• Statistical Advantages of Oblique Randomized Decision Trees and Forests [3.468886360466785]
  Generalization error bounds and convergence rates are obtained for the flexible function class of multi-index models.<n>A lower bound on the risk of axis-aligned Mondrian trees is obtained, proving that these estimators are suboptimal for general ridge functions.
  arXiv  Detail & Related papers  (2024-07-02T17:35:22Z)
• Ensembles of Probabilistic Regression Trees [46.53457774230618]
  Tree-based ensemble methods have been successfully used for regression problems in many applications and research studies. We study ensemble versions of probabilisticregression trees that provide smooth approximations of the objective function by assigningeach observation to each region with respect to a probability distribution.
  arXiv  Detail & Related papers  (2024-06-20T06:51:51Z)
• A Unified Theory of Stochastic Proximal Point Methods without Smoothness [52.30944052987393]
  Proximal point methods have attracted considerable interest owing to their numerical stability and robustness against imperfect tuning. This paper presents a comprehensive analysis of a broad range of variations of the proximal point method (SPPM)
  arXiv  Detail & Related papers  (2024-05-24T21:09:19Z)
• PhyloGFN: Phylogenetic inference with generative flow networks [57.104166650526416]
  We introduce the framework of generative flow networks (GFlowNets) to tackle two core problems in phylogenetics: parsimony-based and phylogenetic inference. Because GFlowNets are well-suited for sampling complex structures, they are a natural choice for exploring and sampling from the multimodal posterior distribution over tree topologies. We demonstrate that our amortized posterior sampler, PhyloGFN, produces diverse and high-quality evolutionary hypotheses on real benchmark datasets.
  arXiv  Detail & Related papers  (2023-10-12T23:46:08Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Random Forest Weighted Local Fréchet Regression with Random Objects [18.128663071848923]
  We propose a novel random forest weighted local Fr'echet regression paradigm.<n>Our first method uses these weights as the local average to solve the conditional Fr'echet mean.<n>Second method performs local linear Fr'echet regression, both significantly improving existing Fr'echet regression methods.
  arXiv  Detail & Related papers  (2022-02-10T09:10:59Z)
• (Decision and regression) tree ensemble based kernels for regression and classification [2.28438857884398]
  Tree based ensembles such as Breiman's random forest (RF) and Gradient Boosted Trees (GBT) can be interpreted as implicit kernel generators. We show that for continuous targets, the RF/GBT kernels are competitive to their respective ensembles in higher dimensional scenarios. We provide the results from real life data sets for regression and classification to show how these insights may be leveraged in practice.
  arXiv  Detail & Related papers  (2020-12-19T16:52:58Z)
• Selective Cascade of Residual ExtraTrees [3.6575928994425735]
  We propose a novel tree-based ensemble method named Selective Cascade of Residual ExtraTrees (SCORE) SCORE draws inspiration from representation learning, incorporates regularized regression with variable selection features, and utilizes boosting to improve prediction and reduce generalization errors. Our computer experiments show that SCORE provides comparable or superior performance in prediction against ExtraTrees, random forest, gradient boosting machine, and neural networks.
  arXiv  Detail & Related papers  (2020-09-29T16:31:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.