Conditional Inference Trees and Forests for Feature Selection

• URL: http://arxiv.org/abs/2607.01417v1
• Date: Wed, 01 Jul 2026 19:23:07 GMT
• Title: Conditional Inference Trees and Forests for Feature Selection
• Abstract summary: Conditional inference trees (CIT) and conditional inference forests (CIF) reduce split-selection bias by testing features before choosing split thresholds.<n>We study CIT and CIF as top-$k$ feature-ranking methods for downstream prediction using real-data benchmarks, runtime ablations, and synthetic feature-recovery experiments.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conditional inference trees (CIT) and conditional inference forests (CIF) reduce split-selection bias by testing features before choosing split thresholds, but repeated permutation tests and threshold searches can make these methods computationally expensive. We study CIT and CIF as top-$k$ feature-ranking methods for downstream prediction using real-data benchmarks, runtime ablations, and synthetic feature-recovery experiments. At a fixed node, if the features and permutation budget do not depend on the node responses, Bonferroni-corrected $+1$ Monte Carlo permutation $p$-values control nodewise rejection under the complete permutation null. CIF ranks 4th among 17 classification methods on 22 datasets and 3rd among 18 regression methods on 8 datasets. With Bonferroni correction held fixed, the CIF runtime ablations indicate that adaptive stopping and the number of thresholds searched have the largest measured effect on runtime: turning off adaptive stopping and using exact threshold search increase fitting time by 4.0--8.4$\times$ and 1.9--10.8$\times$, respectively, while downstream score changes are at most 0.011. Sparse high-$p$ simulations indicate that forest feature sampling can leave informative features out of many split decisions. Overall, the results support CIF as a top-$k$ feature-ranking method in the evaluated downstream prediction benchmarks.

Related papers

• From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion [5.2262942936091905]
  Uniform-state discrete diffusion models update all tokens in parallel while keeping every position revisable.<n>We propose committed reveal sampling (CRS), a training-free sampler that stores selected argmax tokens and inserts them into subsequent model inputs.
  arXiv  Detail & Related papers  (2026-09-01T10:43:24Z)
• Adaptive $k$ Nearest Neighbors Classifier via Granular Ball Computing [60.23863579199315]
  The $k$-Nearest Neighbor(KNN) algorithm is widely used across various tasks.<n>In this paper, an adaptive and efficient KNN approach via granular-ball computing is proposed.
  arXiv  Detail & Related papers  (2026-08-13T07:39:46Z)
• Finding Most Influential Sets [0.0]
  Dinkelbach's method yields an algorithm with $mathcalO(n)$ cost per iteration and finite termination.<n>We show that the method recovers exact MIS that were previously computationally inaccessible.
  arXiv  Detail & Related papers  (2026-06-04T09:24:26Z)
• WARP: A Benchmark for Primal-Dual Warm-Starting of Interior-Point Solvers [0.3058685580689604]
  A growing body of work uses machine learning to predict primal warm-start iterates, reporting reductions of 30-46%.<n>We show that these reported gains rest on an inappropriate evaluation baseline.<n>Experiments establish that the complete primal-dual-barrier state $(x*, *, z*, *)$ reduces IPOPT iterations from 23 to 3.
  arXiv  Detail & Related papers  (2026-05-07T06:19:16Z)
• Efficient Test-Time Inference via Deterministic Exploration of Truncated Decoding Trees [68.04613115686509]
  Self-consistency boosts inference-time performance by sampling multiple reasoning traces in parallel and voting.<n>We propose Distinct Leafion (DLE), a deterministic decoding method that treats truncated sampling as a pruned tree and systematically enumerates distinct leaves instead of sampling with replacement.<n>DLE explores higher-quality reasoning traces, yielding better performance on math, coding, and general reasoning tasks.
  arXiv  Detail & Related papers  (2026-04-22T12:42:03Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• Closing the Approximation Gap of Partial AUC Optimization: A Tale of Two Formulations [121.39938773554523]
  The Area Under the ROC Curve (AUC) is a pivotal evaluation metric in real-world scenarios with both class imbalance and decision constraints.<n>We present two simple instance-wise minimax reformulations to close the approximation gap of PAUC optimization.<n>The resulting algorithms enjoy a linear per-iteration computational complexity w.r.t. the sample size and a convergence rate of $O(-2/3)$ for typical one-way and two-way PAUCs.
  arXiv  Detail & Related papers  (2025-12-01T02:52:33Z)
• Data Selection for ERMs [67.57726352698933]
  We study how well can $mathcalA$ perform when trained on at most $nll N$ data points selected from a population of $N$ points.<n>Our results include optimal data-selection bounds for mean estimation, linear classification, and linear regression.
  arXiv  Detail & Related papers  (2025-04-20T11:26:01Z)
• The Conditioning Bias in Binary Decision Trees and Random Forests and Its Elimination [0.0]
  We investigate the bias introduced by the choice of conditioning operator in the presence of features with lattice characteristics. We propose techniques to eliminate this bias, requiring an additional prediction with decision trees and incurring no cost for random forests.
  arXiv  Detail & Related papers  (2023-12-17T12:56:39Z)
• Simplification of Forest Classifiers and Regressors [1.8275108630751844]
  We study the problem of sharing as many branching conditions of a given forest classifier or regressor as possible. We propose an algorithm for the original problem using an algorithm solving this problem efficiently. The effectiveness of our method is demonstrated through comprehensive experiments using 21 datasets.
  arXiv  Detail & Related papers  (2022-12-14T08:49:02Z)
• Neural Greedy Pursuit for Feature Selection [72.4121881681861]
  We propose a greedy algorithm to select $N$ important features among $P$ input features for a non-linear prediction problem. We use neural networks as predictors in the algorithm to compute the loss.
  arXiv  Detail & Related papers  (2022-07-19T16:39:16Z)
• Conditional Uncorrelation and Efficient Non-approximate Subset Selection in Sparse Regression [72.84177488527398]
  We consider sparse regression from the view of correlation, and propose the formula of conditional uncorrelation. By the proposed method, the computational complexity is reduced from $O(frac16k3+mk2+mkd)$ to $O(frac16k3+frac12mk2)$ for each candidate subset in sparse regression.
  arXiv  Detail & Related papers  (2020-09-08T20:32:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.