RobustModelMaker: Coupling Bootstrap Stability Selection with Leakage-Safe Nested Cross-Validation for Scientific Machine Learning

• URL: http://arxiv.org/abs/2606.01566v1
• Date: Mon, 01 Jun 2026 02:11:23 GMT
• Title: RobustModelMaker: Coupling Bootstrap Stability Selection with Leakage-Safe Nested Cross-Validation for Scientific Machine Learning
• Abstract summary: RobustModelMaker is a Python framework that bootstraps stability selection with strict nested cross-validation.<n>It produces a stability-tested feature subset together with a leakage-safe performance estimate.<n>It supports nine algorithms across binary classification, multiclass classification, and regression.
• Score: 2.0052993723676895
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Small-to-medium scientific datasets place machine learning pipelines under two compounding pressures. Single-run feature selection produces feature sets that change substantially under small perturbations of the training data, and any procedure that uses the same data for selection, tuning, and evaluation produces optimistically biased performance estimates. The two failure modes are routinely treated as separable, but in the regimes where scientific data live, they interact: an unstable selection inflates the variance of an already-optimistic score, and standard remedies for one rarely address the other. RobustModelMaker is a Python framework that couples bootstrap stability selection with strict nested cross-validation, performs all preprocessing and selection inside each fold, and produces a stability-tested feature subset together with a leakage-safe performance estimate. The framework supports nine algorithms across binary classification, multiclass classification, and regression. Behaviour is verified by a deterministic test suite spanning unit, performance, and reproducibility checks on three real scientific datasets comparing to three alternative selectors (ANOVA F-test, recursive feature elimination with cross-validation, and Boruta) on both predictive score and a Jaccard measure of selection stability. RobustModelMaker is competitive in score with the best alternative selector on each dataset, and occupies a position on the joint score-stability frontier that none of the alternatives match across all three task types. Two example applications, ovarian cancer biomarker discovery from the PLCO Trial and critical-temperature regression on the UCI Superconductivity Data, illustrate how the framework is used in practice and what trade-offs become visible when stability is treated as a first-class deliverable rather than an emergent property.

Related papers

• Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data [73.19814469893088]
  Digital twins and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems.<n>This paper addresses whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible.<n>Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing.
  arXiv  Detail & Related papers  (2026-08-28T07:05:38Z)
• Tail-Aware Adaptive-k: Query-Adaptive Context Selection for Retrieval-Augmented Generation [0.9786690381850356]
  We propose Tail-Aware Adaptive-k(TAA-k), a training-free framework that operationalizes Extreme Value Theory (EVT) through a localized validation strategy.<n>Under mild monotone likelihood ratio assumptions, TAA-k yields a stable, query-adaptive cutoff corresponding to the earliest noise-dominated position.<n>Experiments on WebQuestions, 2WikiMultiHopQA, and MuSiQue demonstrate that TAA-k achieves near-oracle retrieval quality (F1 within 2-3% of oracle) with orders-of-magnitude efficiency gains over global EVT methods.
  arXiv  Detail & Related papers  (2026-06-10T10:37:19Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• From Invariant Representations to Invariant Data: Provable Robustness to Spurious Correlations via Noisy Counterfactual Matching [14.679039741833305]
  Models that learn spurious correlations from training data often fail when deployed in new environments.<n>We propose a data-centric alternative that shifts the focus from learning invariant representations to leveraging invariant data pairs.<n>Noisy Counterfactual Matching improves robustness by leveraging even a small number of emphnoisy counterfactual pairs.
  arXiv  Detail & Related papers  (2025-05-30T17:42:32Z)
• Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization [4.984656106595651]
  This paper presents OptCS, a framework that allows valid statistical testing (selection) after flexible data-driven model optimization.<n>We introduce general conditions under which OptCS constructs valid conformal p-values despite substantial data reuse.<n>We propose three FDR-controlling procedures, each optimizing the models differently.
  arXiv  Detail & Related papers  (2024-11-27T01:40:50Z)
• Improving Bias Correction Standards by Quantifying its Effects on Treatment Outcomes [54.18828236350544]
  Propensity score matching (PSM) addresses selection biases by selecting comparable populations for analysis. Different matching methods can produce significantly different Average Treatment Effects (ATE) for the same task, even when meeting all validation criteria. To address this issue, we introduce a novel metric, A2A, to reduce the number of valid matches.
  arXiv  Detail & Related papers  (2024-07-20T12:42:24Z)
• SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities [50.6382396309597]
  Unsupervised Domain Adaptation (DA) consists of adapting a model trained on a labeled source domain to perform well on an unlabeled target domain with some data distribution shift.<n>We present a complete and fair evaluation of existing shallow algorithms, including reweighting, mapping, and subspace alignment.<n>Our benchmark highlights the importance of realistic validation and provides practical guidance for real-life applications.
  arXiv  Detail & Related papers  (2024-07-16T12:52:29Z)
• Causal Feature Selection via Transfer Entropy [59.999594949050596]
  Causal discovery aims to identify causal relationships between features with observational data. We introduce a new causal feature selection approach that relies on the forward and backward feature selection procedures. We provide theoretical guarantees on the regression and classification errors for both the exact and the finite-sample cases.
  arXiv  Detail & Related papers  (2023-10-17T08:04:45Z)
• Contrastive Neural Ratio Estimation for Simulation-based Inference [15.354874711988662]
  Likelihood-to-evidence ratio estimation is usually cast as either a binary (NRE-A) or a multiclass (NRE-B) classification task. In contrast to the binary classification framework, the current formulation of the multiclass version has an intrinsic and unknown bias term. We propose a multiclass framework free from the bias inherent to NRE-B at optimum, leaving us in the position to run diagnostics that practitioners depend on.
  arXiv  Detail & Related papers  (2022-10-11T00:12:51Z)
• Loss-guided Stability Selection [0.0]
  It is well-known that model selection procedures like the Lasso or Boosting tend to overfit on real data. Standard Stability Selection is based on a global criterion, namely the per-family error rate. We propose a Stability Selection variant which respects the chosen loss function via an additional validation step.
  arXiv  Detail & Related papers  (2022-02-10T11:20:25Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.