Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well

• URL: http://arxiv.org/abs/2606.12552v1
• Date: Wed, 10 Jun 2026 18:03:19 GMT
• Title: Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well
• Abstract summary: We show that cross-validation improves markedly confidence when evaluating and comparing learning algorithm performances.<n>We introduce the concept of sample gain, which quantifies the virtual data augmentation achieved by using multiple cross-validation splits.
• Score: 15.198198886832595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern machine learning progresses through empirical work, benchmarking new methods to evaluate relative performance. However, the statistical variability inherent to evaluation - exacerbated by the stochastic nature of many algorithms - often makes performance estimation unreliable due to the limited test samples available, leading to a validation crisis in which genuine advances are difficult to discern. In this work, we show that cross-validation improves markedly confidence when evaluating and comparing learning algorithm performances. We introduce the concept of sample gain, which quantifies the virtual data augmentation achieved by using multiple cross-validation splits to reduce benchmarking variance. Experiments on both synthetic and real-world datasets (histopathologic scans and NLP fine-tuning) demonstrate that multiple splits can substantially improve the reliability and stability of performance estimates, with diminishing returns often setting in later than expected. We also introduce a procedure to dynamically early-stop cross-validation by estimating from the first few folds if subsequent folds will bring large sample gains. Our findings highlight the value of pushing cross-validation on available samples to achieve robust and reliable benchmarking.

Related papers

• Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison [17.732982117200425]
  Generalist robot manipulation policies are becoming increasingly capable, but are limited in evaluation to a small number of hardware rollouts.<n>This work presents a novel framework for robot policy comparison that is sample-efficient, statistically rigorous, and applicable to a broad set of evaluation metrics used in practice.
  arXiv  Detail & Related papers  (2026-03-13T21:47:48Z)
• Sharp Convergence Rates for Masked Diffusion Models [53.117058231393834]
  We develop a total-variation based analysis for the Euler method that overcomes limitations.<n>Our results relax assumptions on score estimation, improve parameter dependencies, and establish convergence guarantees.<n>Overall, our analysis introduces a direct TV-based error decomposition along the CTMC trajectory and a decoupling-based path-wise analysis for FHS.
  arXiv  Detail & Related papers  (2026-02-26T00:47:51Z)
• Modest-Align: Data-Efficient Alignment for Vision-Language Models [67.48633659305592]
  Cross-modal alignment models often suffer from overconfidence and degraded performance when operating in resource-constrained settings.<n>We propose Modest-Align, a lightweight alignment framework designed for robustness and efficiency.<n>Our method offers a practical and scalable solution for cross-modal alignment in real-world, low-resource scenarios.
  arXiv  Detail & Related papers  (2025-10-24T16:11:10Z)
• Irredundant $k$-Fold Cross-Validation [0.0]
  In traditional k-fold cross-validation, each instance is used ($k-1$) times for training and once for testing, leading to redundancy.<n>We introduce Irredundant $k$-fold cross-validation, a novel method that guarantees each instance is used exactly once for training and once for testing.
  arXiv  Detail & Related papers  (2025-07-26T19:59:37Z)
• DOTA: Distributional Test-Time Adaptation of Vision-Language Models [69.41389326333771]
  Vision-language foundation models can be unreliable when significant distribution gaps exist between training and test data.<n>We propose DOTA (DistributiOnal Test-time Adaptation), a simple yet effective method addressing this limitation.<n>This distribution-centric approach enables the model to continually learn and adapt to the deployment environment.
  arXiv  Detail & Related papers  (2024-09-28T15:03:28Z)
• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [65.21599711087538]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample.<n>Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications.<n>We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Bootstrapping the Cross-Validation Estimate [5.816215734815304]
  Cross-validation is a widely used technique for evaluating the performance of prediction models.<n>This paper proposes a fast bootstrap method that quickly estimates the standard error of the cross-validation estimate.
  arXiv  Detail & Related papers  (2023-07-01T07:50:54Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• Fast calculation of Gaussian Process multiple-fold cross-validation residuals and their covariances [0.6091702876917281]
  We generalize fast leave-one-out formulae to multiple-fold cross-validation. We highlight the covariance structure of cross-validation residuals in both Simple and Universal Kriging frameworks. Our results enable fast multiple-fold cross-validation and have direct consequences in model diagnostics.
  arXiv  Detail & Related papers  (2021-01-08T17:02:37Z)
• Modeling Score Distributions and Continuous Covariates: A Bayesian Approach [8.772459063453285]
  We develop a generative model of the match and non-match score distributions over continuous covariates. We use mixture models to capture arbitrary distributions and local basis functions. Three experiments demonstrate the accuracy and effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-09-21T02:41:20Z)
• Cross-validation Confidence Intervals for Test Error [83.67415139421448]
  This work develops central limit theorems for crossvalidation and consistent estimators of its variance under weak stability conditions on the learning algorithm. Results are the first of their kind for the popular choice of leave-one-out cross-validation.
  arXiv  Detail & Related papers  (2020-07-24T17:40:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.