Intra-tree Column Subsampling Hinders XGBoost Learning of Ratio-like Interactions

• URL: http://arxiv.org/abs/2601.08121v1
• Date: Tue, 13 Jan 2026 01:27:01 GMT
• Title: Intra-tree Column Subsampling Hinders XGBoost Learning of Ratio-like Interactions
• Abstract summary: We study whether intra-tree column subsampling in XGBoost makes that synthesis harder.<n>Across both processes, intra-tree column subsampling reduces test PR-AUC in the primitives-only setting.<n> Practically, if ratio-like structure is plausible, either avoid intra-tree subsampling or include the intended ratio features.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many applied problems contain signal that becomes clear only after combining multiple raw measurements. Ratios and rates are common examples. In gradient boosted trees, this combination is not an explicit operation: the model must synthesize it through coordinated splits on the component features. We study whether intra-tree column subsampling in XGBoost makes that synthesis harder. We use two synthetic data generating processes with cancellation-style structure. In both, two primitive features share a strong nuisance factor, while the target depends on a smaller differential factor. A log ratio cancels the nuisance and isolates the signal. We vary colsample_bylevel and colsample_bynode over s in {0.4, 0.6, 0.8, 0.9}, emphasizing mild subsampling (s >= 0.8). A control feature set includes the engineered ratio, removing the need for synthesis. Across both processes, intra-tree column subsampling reduces test PR-AUC in the primitives-only setting. In the main process the relative decrease reaches 54 percent when both parameters are set to 0.4. The effect largely disappears when the engineered ratio is present. A path-based co-usage metric drops in the same cells where performance deteriorates. Practically, if ratio-like structure is plausible, either avoid intra-tree subsampling or include the intended ratio features.

Related papers

• Shared Symbolic Backbones for Physically Consistent Multi-Output Symbolic Regression [10.085152540674164]
  We present a neuro-evolutionary symbolic regression method for coupled multi-output systems.<n>The method searches for a shared symbolic backbone.<n>The proposed framework, rather than a general purpose predictor, is a structured shared-mechanism extractor.
  arXiv  Detail & Related papers  (2026-07-29T06:47:36Z)
• Significance-First Splitting: Aligning Treatment Heterogeneity Detection with Honest Estimation [0.0]
  Estimating heterogeneous treatment effects (CATE) requires simultaneously detecting effect modification and quantifying estimation uncertainty.<n>Existing tree-based methods make an uneasy trade-off: significance-based approaches identify subgroup interactions directly but lack valid inference.<n>We introduce a hybrid algorithm that fuses significance-based splitting with honest sample-splitting and cross-validation.
  arXiv  Detail & Related papers  (2026-07-04T19:50:35Z)
• Physics-Aware Auxiliary Losses Improve Out-of-Distribution Generalization of a GNN Synthesizability Filter [1.1986159241687198]
  We ask whether cheap, closed-form physical priors, used as auxiliary supervision on a graph neural network (GNN), improve OOD generalization.<n>We add two auxiliary losses to a GINE backbone: a topological complexity regression supervised by the Bertz index, and a strain-energy soft penalty supervised by MMFF94 force-field energy.<n>All three physics-aware variants give a small but statistically significant OOD improvement over the baseline.
  arXiv  Detail & Related papers  (2026-06-10T20:21:56Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Stop Marginalizing My Dreams: Model Inversion via Laplace Kernel for Continual Learning [8.71890550161683]
  Data-free continual learning relies on model inversion to synthesize pseudo-samples and forgetting catastrophic forgetting.<n>Existing inversion methods are limited by a simplifying assumption.<n>We show that modeling feature dependencies is a key ingredient for effective DFCIL.
  arXiv  Detail & Related papers  (2026-05-12T08:59:27Z)
• Unbiased Gradient Estimation for Event Binning via Functional Backpropagation [64.88399635309918]
  We propose a novel framework for unbiased gradient estimation of arbitrary binning functions by synthesizing weak derivatives during backpropagation.<n>We achieve 9.4% lower EPE in self-supervised optical flow, and 5.1% lower RMS error in SLAM, demonstrating broad benefits for event-based visual perception.
  arXiv  Detail & Related papers  (2026-02-13T04:05:03Z)
• Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting [18.562483381753804]
  We present the Hinge Regression Tree (HRT), which reframes each split as a non-linear least-squares problem over two linear predictors.<n>We analyze this node-level optimization and, for a backtracking line-search variant, prove that the local objective decreases monotonically and converges.<n>We show on synthetic and real-world benchmarks that HRT matches or outperforms single-tree baselines with more compact structures.
  arXiv  Detail & Related papers  (2026-02-05T06:49:01Z)
• Ensemble Threshold Calibration for Stable Sensitivity Control [0.0]
  We present an end-to-end framework that achieves exact recall with sub-percent variance over tens of millions of geometry pairs.<n>Our approach consistently hits a recall target within a small error, decreases redundant verifications relative to other calibrations, and runs end-to-end on a single TPU v3 core.
  arXiv  Detail & Related papers  (2025-10-02T15:22:28Z)
• STRIDE: Subset-Free Functional Decomposition for XAI in Tabular Settings [0.0]
  We present STRIDE, a framework that answers "what" features are important but fails to reveal "how" they interact.<n>We also introduce "component surgery", a diagnostic that isolates a learned interaction and quantifies its contribution.
  arXiv  Detail & Related papers  (2025-09-11T00:19:53Z)
• CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective [48.99488315273868]
  We propose a contrastive knowledge distillation framework that achieves sample-wise logit alignment while preserving semantic consistency.<n>Our approach transfers "dark knowledge" through teacher-student contrastive alignment at the sample level.<n>We conduct comprehensive experiments across three benchmark datasets, including the CIFAR-100, ImageNet-1K, and MS COCO datasets.
  arXiv  Detail & Related papers  (2024-04-22T11:52:40Z)
• Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth [83.15263499262824]
  We prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. We show how to improve upon Gaussian performance for the compression of sparse data by adding a denoising function to a shallow architecture. We validate our findings on image datasets, such as CIFAR-10 and MNIST.
  arXiv  Detail & Related papers  (2024-02-07T16:32:29Z)
• Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion [85.54515118077825]
  This paper proposes a linear diffusion model (LinDiff) based on an ordinary differential equation to simultaneously reach fast inference and high sample quality. To reduce computational complexity, LinDiff employs a patch-based processing approach that partitions the input signal into small patches. Our model can synthesize speech of a quality comparable to that of autoregressive models with faster synthesis speed.
  arXiv  Detail & Related papers  (2023-06-09T07:02:43Z)
• Estimating Higher-Order Mixed Memberships via the $\ell_{2,\infty}$ Tensor Perturbation Bound [8.521132000449766]
  We propose the tensor mixed-membership blockmodel, a generalization of the tensor blockmodel. We establish the identifiability of our model and propose a computationally efficient estimation procedure. We apply our methodology to real and simulated data, demonstrating some effects not identifiable from the model with discrete community memberships.
  arXiv  Detail & Related papers  (2022-12-16T18:32:20Z)
• Active-LATHE: An Active Learning Algorithm for Boosting the Error Exponent for Learning Homogeneous Ising Trees [75.93186954061943]
  We design and analyze an algorithm that boosts the error exponent by at least 40% when $rho$ is at least $0.8$. Our analysis hinges on judiciously exploiting the minute but detectable statistical variation of the samples to allocate more data to parts of the graph.
  arXiv  Detail & Related papers  (2021-10-27T10:45:21Z)
• Analyzing and Mitigating Interference in Neural Architecture Search [96.60805562853153]
  We investigate the interference issue by sampling different child models and calculating the gradient similarity of shared operators. Inspired by these two observations, we propose two approaches to mitigate the interference. Our searched architecture outperforms RoBERTa$_rm base$ by 1.1 and 0.6 scores and ELECTRA$_rm base$ by 1.6 and 1.1 scores on the dev and test set of GLUE benchmark.
  arXiv  Detail & Related papers  (2021-08-29T11:07:46Z)
• Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers [59.06169363181417]
  Predict then Interpolate (PI) is an algorithm for learning correlations that are stable across environments. We prove that by interpolating the distributions of the correct predictions and the wrong predictions, we can uncover an oracle distribution where the unstable correlation vanishes.
  arXiv  Detail & Related papers  (2021-05-26T15:37:48Z)
• A better method to enforce monotonic constraints in regression and classification trees [0.0]
  We present two new ways of enforcing monotone constraints in regression and classification trees. One yields better results than the current LightGBM, and has a similar computation time. The other one yields even better results, but is much slower than the current LightGBM.
  arXiv  Detail & Related papers  (2020-11-02T14:04:21Z)
• The Tree Ensemble Layer: Differentiability meets Conditional Computation [8.40843862024745]
  We introduce a new layer for neural networks composed of an ensemble of differentiable decision trees (a.k.a. soft trees) Differentiable trees demonstrate promising results in the literature, but are typically slow in training and inference as they do not support conditional computation. We develop specialized forward and backward propagation algorithms that exploit sparsity.
  arXiv  Detail & Related papers  (2020-02-18T18:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.