Bootstrap Bias Corrected Cross Validation applied to Super Learning

• URL: http://arxiv.org/abs/2003.08342v1
• Date: Wed, 18 Mar 2020 17:12:42 GMT
• Title: Bootstrap Bias Corrected Cross Validation applied to Super Learning
• Authors: Krzysztof Mnich and Agnieszka Kitlas Goli\'nska and Aneta Polewko-Klim and Witold R. Rudnicki
• Abstract summary: Super learner algorithm can be applied to combine results of multiple base learners to improve quality of predictions. Tests were performed on artificial data sets of diverse size and on seven real, biomedical data sets. The resampling method, called Bootstrap Bias Correction, proved to be a reasonably precise and very cost-efficient alternative for nested cross validation.
• Score: 0.3670422696827526
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Super learner algorithm can be applied to combine results of multiple base learners to improve quality of predictions. The default method for verification of super learner results is by nested cross validation. It has been proposed by Tsamardinos et al., that nested cross validation can be replaced by resampling for tuning hyper-parameters of the learning algorithms. We apply this idea to verification of super learner and compare with other verification methods, including nested cross validation. Tests were performed on artificial data sets of diverse size and on seven real, biomedical data sets. The resampling method, called Bootstrap Bias Correction, proved to be a reasonably precise and very cost-efficient alternative for nested cross validation.

Related papers

• PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering [71.15346406323827]
  We introduce PRIME, a benchmark for evaluating verifiers on Process-Outcome Alignment verification.<n>We find that current verifiers frequently fail to detect derivation flaws.<n>We propose a process-aware RLVR training paradigm utilizing verifiers selected via PRIME.
  arXiv  Detail & Related papers  (2026-02-12T04:45:01Z)
• Irredundant k-Fold Cross-Validation [0.0]
  In traditional k-fold cross-validation, each instance is used ($k!-!1$) times for training and once for testing, leading to redundancy.<n>We introduce Irredundant $k$--fold cross-validation, a novel method that guarantees each instance is used exactly once for training and once for testing.
  arXiv  Detail & Related papers  (2025-07-26T19:59:37Z)
• A New Flexible Train-Test Split Algorithm, an approach for choosing among the Hold-out, K-fold cross-validation, and Hold-out iteration [0.0]
  This study focuses on improving the accuracy of ML algorithms across three different datasets. By modifying parameters like test size, Random State, and 'k' values, we were able to improve accuracy assessment. This study challenges the universality of K values in K-Fold Cross Validation and suggests a 10% test size and 90% training size for better outcomes.
  arXiv  Detail & Related papers  (2025-01-11T09:42:13Z)
• Crafting Distribution Shifts for Validation and Training in Single Source Domain Generalization [9.323285246024518]
  Single-source domain generalization attempts to learn a model on a source domain and deploy it to unseen target domains. Standard practice of validation on the training distribution does not accurately reflect the model's generalization ability. We construct an independent validation set by transforming source domain images with a comprehensive list of augmentations.
  arXiv  Detail & Related papers  (2024-09-29T20:52:50Z)
• Systematic comparison of semi-supervised and self-supervised learning for medical image classification [12.977356726499735]
  In typical medical image classification problems, labeled data is scarce while unlabeled data is more available. Recent methods from both directions have reported significant gains on traditional benchmarks. Our study compares 13 representative semi- and self-supervised methods to strong labeled-set-only baselines on 4 medical datasets.
  arXiv  Detail & Related papers  (2023-07-18T01:31:47Z)
• Dense FixMatch: a simple semi-supervised learning method for pixel-wise prediction tasks [68.36996813591425]
  We propose Dense FixMatch, a simple method for online semi-supervised learning of dense and structured prediction tasks. We enable the application of FixMatch in semi-supervised learning problems beyond image classification by adding a matching operation on the pseudo-labels. Dense FixMatch significantly improves results compared to supervised learning using only labeled data, approaching its performance with 1/4 of the labeled samples.
  arXiv  Detail & Related papers  (2022-10-18T15:02:51Z)
• CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
  Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
  arXiv  Detail & Related papers  (2022-06-01T03:02:07Z)
• Smooth-Reduce: Leveraging Patches for Improved Certified Robustness [100.28947222215463]
  We propose a training-free, modified smoothing approach, Smooth-Reduce. Our algorithm classifies overlapping patches extracted from an input image, and aggregates the predicted logits to certify a larger radius around the input. We provide theoretical guarantees for such certificates, and empirically show significant improvements over other randomized smoothing methods.
  arXiv  Detail & Related papers  (2022-05-12T15:26:20Z)
• A New Approach to Multilabel Stratified Cross Validation with Application to Large and Sparse Gene Ontology Datasets [0.0]
  We show a weakness in an evaluation metric widely used in literature. We present improved versions of this metric and a general method, optisplit, for optimising cross validations splits. We show that optisplit produces better cross validation splits than the existing methods and that it is fast enough to be used on big Gene Ontology datasets.
  arXiv  Detail & Related papers  (2021-09-03T10:34:22Z)
• Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density [125.64297244986552]
  We propose an unsupervised validation criterion that measures the density of soft neighborhoods by computing the entropy of the similarity distribution between points. Our criterion is simpler than competing validation methods, yet more effective.
  arXiv  Detail & Related papers  (2021-08-24T17:41:45Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Posterior Re-calibration for Imbalanced Datasets [33.379680556475314]
  Neural Networks can perform poorly when the training label distribution is heavily imbalanced. We derive a post-training prior rebalancing technique that can be solved through a KL-divergence based optimization. Our results on six different datasets and five different architectures show state of art accuracy.
  arXiv  Detail & Related papers  (2020-10-22T15:57:14Z)
• Cross-validation Confidence Intervals for Test Error [83.67415139421448]
  This work develops central limit theorems for crossvalidation and consistent estimators of its variance under weak stability conditions on the learning algorithm. Results are the first of their kind for the popular choice of leave-one-out cross-validation.
  arXiv  Detail & Related papers  (2020-07-24T17:40:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.