When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

• URL: http://arxiv.org/abs/2608.16147v1
• Date: Mon, 17 Aug 2026 05:58:33 GMT
• Title: When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification
• Abstract summary: Class-imbalance handling is routinely evaluated on a single benchmark dataset.<n>Under a leakage-free nested crossvalidation protocol, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021.<n> Threshold-tuning benefit is non-monotonic in the imbalance ratio.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.

Related papers

• Replicating TRACE: A Practitioner's Guide to Its Threshold and Particle Budget [0.10923877073891443]
  TRACE reads causal graphs out of a pretrained autoregressive sequence model.<n>Mean per-sequence F1 against exact interventional truth reaches 0.90-0.91 at vocabulary size 1000.
  arXiv  Detail & Related papers  (2026-09-01T11:48:16Z)
• The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models [22.461983628330277]
  Many few-shot adaptation methods classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features.<n>We ask what is the right ratio, can it be estimated without validation data, and is finding it where the performance is?<n>Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity.<n> validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average.
  arXiv  Detail & Related papers  (2026-08-23T15:02:33Z)
• Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection [0.0]
  Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings.<n>We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it work carries more variance than the null it is tested against.
  arXiv  Detail & Related papers  (2026-08-12T23:27:20Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Why Does the Future Branch? Identifiable Closure Tests for Stochastic Physical World Models [0.0]
  ClosurePairs crosses compatible microstates with repeated disturbances.<n>It recovers source attribution at unchanged likelihood.<n>It reduces equal-budget decomposition error in a nonlinear interaction benchmark.
  arXiv  Detail & Related papers  (2026-08-01T11:07:24Z)
• Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion [46.262619407930266]
  We train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings.<n>The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline.<n>We then investigate whether LLM-assisted labeling can extend the gold dataset.
  arXiv  Detail & Related papers  (2026-07-28T10:59:04Z)
• K-ABENA: K-Adaptive Backpropagation with Error-based N-exclusion Algorithm : (Compensated Loss-Based Sample Exclusion with Unbiased Gradient Estimation) [0.0]
  K-ABENAK-Adaptive Backpropagation with Error-based N-exclusion is a selective computation framework.<n>It reduces per-it computation training cost by excluding a fraction of low-loss ("minor") observations.<n>It collapses to 0.386 accuracy under label limitation (baseline: 0.832) and to 0.53 AUC under extreme imbalance.
  arXiv  Detail & Related papers  (2026-07-07T06:58:51Z)
• PAIR-CI: Calibrated Conditional Independence Testing for Causal Discovery with Incomplete Data [0.0]
  We introduce PAIR-CI, a nonparametric conditional independence (CI) test that restores calibration by integrating multiple imputation directly into the inferential procedure.<n>A provably consistent variance estimator jointly accounts for uncertainty arising from cross-validation and multiple imputation.
  arXiv  Detail & Related papers  (2026-05-06T12:34:37Z)
• One Sample is Enough to Make Conformal Prediction Robust [53.78604391939934]
  We show that conformal prediction attains some robustness even with a forward pass on a single randomly perturbed input.<n>Our approach returns robust sets with smaller average set size compared to SOTA methods which use many (e.g. around 100) passes per input.
  arXiv  Detail & Related papers  (2025-06-19T19:14:25Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability.<n>We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• ADT-SSL: Adaptive Dual-Threshold for Semi-Supervised Learning [68.53717108812297]
  Semi-Supervised Learning (SSL) has advanced classification tasks by inputting both labeled and unlabeled data to train a model jointly. This paper proposes an Adaptive Dual-Threshold method for Semi-Supervised Learning (ADT-SSL) Experimental results show that the proposed ADT-SSL achieves state-of-the-art classification accuracy.
  arXiv  Detail & Related papers  (2022-05-21T11:52:08Z)
• FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning [46.95063831057502]
  We propose emphFreeMatch to define and adjust the confidence threshold in a self-adaptive manner according to the model's learning status. FreeMatch achieves textbf5.78%, textbf13.59%, and textbf1.28% error rate reduction over the latest state-of-the-art method FlexMatch on CIFAR-10 with 1 label per class.
  arXiv  Detail & Related papers  (2022-05-15T10:07:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.