Jacobian Aligned Random Forests

• URL: http://arxiv.org/abs/2512.08306v1
• Date: Tue, 09 Dec 2025 07:08:04 GMT
• Title: Jacobian Aligned Random Forests
• Authors: Sarwesh Rauniyar,
• Abstract summary: We present Jacobian-Aligned Random Forests (JARF) as an alternative to oblique forests for axis-aligned decision trees.<n>We show that JARF consistently improves axis-aligned forests and often matches or surpasses baselines while improving training time.<n>Our experimental results and theoretical analysis together indicate that supervised preconditioning can recover much of the accuracy of oblique forests while retaining the simplicity and robustness of axis-aligned trees.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Axis-aligned decision trees are fast and stable but struggle on datasets with rotated or interaction-dependent decision boundaries, where informative splits require linear combinations of features rather than single-feature thresholds. Oblique forests address this with per-node hyperplane splits, but at added computational cost and implementation complexity. We propose a simple alternative: JARF, Jacobian-Aligned Random Forests. Concretely, we first fit an axis-aligned forest to estimate class probabilities or regression outputs, compute finite-difference gradients of these predictions with respect to each feature, aggregate them into an expected Jacobian outer product that generalizes the expected gradient outer product (EGOP), and use it as a single global linear preconditioner for all inputs. This supervised preconditioner applies a single global rotation of the feature space, then hands the transformed data back to a standard axis-aligned forest, preserving off-the-shelf training pipelines while capturing oblique boundaries and feature interactions that would otherwise require many axis-aligned splits to approximate. The same construction applies to any model that provides gradients, though we focus on random forests and gradient-boosted trees in this work. On tabular classification and regression benchmarks, this preconditioning consistently improves axis-aligned forests and often matches or surpasses oblique baselines while improving training time. Our experimental results and theoretical analysis together indicate that supervised preconditioning can recover much of the accuracy of oblique forests while retaining the simplicity and robustness of axis-aligned trees.

Related papers

• TreeGrad-Ranker: Feature Ranking via $O(L)$-Time Gradients for Decision Trees [73.0940890296463]
  probabilistic values are used to rank features for explaining local predicted values of decision trees.<n>TreeGrad computes the gradients of the multilinear extension of the joint objective in $O(L)$ time for decision trees with $L$ leaves.<n>TreeGrad-Ranker aggregates the gradients while optimizing the joint objective to produce feature rankings.<n>TreeGrad-Shap is a numerically stable algorithm for computing Beta Shapley values with integral parameters.
  arXiv  Detail & Related papers  (2026-02-12T06:17:12Z)
• Partition Trees: Conditional Density Estimation over General Outcome Spaces [46.1988967916659]
  We propose Partition Trees, a tree-based framework for conditional density estimation over general outcome spaces.<n>Our approach models conditional distributions as piecewise-constant densities on data adaptive partitions and learns trees by directly minimizing conditional negative log-likelihood.
  arXiv  Detail & Related papers  (2026-02-03T22:12:30Z)
• Lassoed Forests: Random Forests with Adaptive Lasso Post-selection [36.24615773895282]
  We show in theory that the relative performance of two methods, standard and Lasso-weighted random forests, depends on the signal-to-noise ratio.<n>We propose a unified framework to combine random forests and Lasso selection by applying adaptive weighting.
  arXiv  Detail & Related papers  (2025-11-10T04:39:09Z)
• A Powerful Random Forest Featuring Linear Extensions (RaFFLE) [1.2233362977312945]
  RaFFLE is a novel framework that integrates PILOT trees as base learners within a random forest ensemble.<n> PILOT trees combine the computational efficiency of traditional decision trees with the flexibility of linear model trees.<n>RaFFLE proves to be a versatile tool for tackling a wide variety of regression problems.
  arXiv  Detail & Related papers  (2025-02-14T14:22:51Z)
• Statistical Advantages of Oblique Randomized Decision Trees and Forests [3.468886360466785]
  Generalization error bounds and convergence rates are obtained for the flexible function class of multi-index models.<n>A lower bound on the risk of axis-aligned Mondrian trees is obtained, proving that these estimators are suboptimal for general ridge functions.
  arXiv  Detail & Related papers  (2024-07-02T17:35:22Z)
• Ensembles of Probabilistic Regression Trees [46.53457774230618]
  Tree-based ensemble methods have been successfully used for regression problems in many applications and research studies. We study ensemble versions of probabilisticregression trees that provide smooth approximations of the objective function by assigningeach observation to each region with respect to a probability distribution.
  arXiv  Detail & Related papers  (2024-06-20T06:51:51Z)
• Adaptive Split Balancing for Optimal Random Forest [8.916614661563893]
  We propose a new random forest algorithm that constructs the trees using a novel adaptive split-balancing method. Our method achieves optimality in simple, smooth scenarios while adaptively learning the tree structure from the data.
  arXiv  Detail & Related papers  (2024-02-17T09:10:40Z)
• Distributional Adaptive Soft Regression Trees [0.0]
  This article proposes a new type of a distributional regression tree using a multivariate soft split rule. One great advantage of the soft split is that smooth high-dimensional functions can be estimated with only one tree. We show by means of extensive simulation studies that the algorithm has excellent properties and outperforms various benchmark methods.
  arXiv  Detail & Related papers  (2022-10-19T08:59:02Z)
• Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes [38.157373733083894]
  This paper introduces a framework designed to accurately predict piecewise linear mappings of arbitrary meshes via a neural network. The framework is based on reducing the neural aspect to a prediction of a matrix for a single point, conditioned on a global shape descriptor. By operating in the intrinsic gradient domain of each individual mesh, it allows the framework to predict highly-accurate mappings.
  arXiv  Detail & Related papers  (2022-05-05T19:51:13Z)
• Distributional Gradient Boosting Machines [77.34726150561087]
  Our framework is based on XGBoost and LightGBM. We show that our framework achieves state-of-the-art forecast accuracy.
  arXiv  Detail & Related papers  (2022-04-02T06:32:19Z)
• Minimax Rates for High-Dimensional Random Tessellation Forests [0.0]
  Mondrian forests is the first class of random forests for which minimax rates were obtained in arbitrary dimension. We show that a large class of random forests with general split directions also achieve minimax optimal convergence rates in arbitrary dimension.
  arXiv  Detail & Related papers  (2021-09-22T06:47:38Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)
• Growing Deep Forests Efficiently with Soft Routing and Learned Connectivity [79.83903179393164]
  This paper further extends the deep forest idea in several important aspects. We employ a probabilistic tree whose nodes make probabilistic routing decisions, a.k.a., soft routing, rather than hard binary decisions. Experiments on the MNIST dataset demonstrate that our empowered deep forests can achieve better or comparable performance than [1],[3].
  arXiv  Detail & Related papers  (2020-12-29T18:05:05Z)
• Residual Correlation in Graph Neural Network Regression [39.54530450932135]
  We show that conditional independence assumption severely limits predictive power. We address this problem with an interpretable and efficient framework. Our framework achieves substantially higher accuracy than competing baselines.
  arXiv  Detail & Related papers  (2020-02-19T16:32:54Z)
• Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation [77.7420231319632]
  We adapt contextual generation of categorical sequences to a policy gradient estimator, which evaluates a set of correlated Monte Carlo (MC) rollouts for variance control. We also demonstrate the use of correlated MC rollouts for binary-tree softmax models, which reduce the high generation cost in large vocabulary scenarios.
  arXiv  Detail & Related papers  (2019-12-31T03:01:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.