Cross-validation Confidence Intervals for Test Error

• URL: http://arxiv.org/abs/2007.12671v2
• Date: Sat, 31 Oct 2020 17:24:26 GMT
• Title: Cross-validation Confidence Intervals for Test Error
• Authors: Pierre Bayle, Alexandre Bayle, Lucas Janson, Lester Mackey
• Abstract summary: This work develops central limit theorems for crossvalidation and consistent estimators of its variance under weak stability conditions on the learning algorithm. Results are the first of their kind for the popular choice of leave-one-out cross-validation.
• Score: 83.67415139421448
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work develops central limit theorems for cross-validation and consistent estimators of its asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for $k$-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller $k$-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our real-data experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.

Related papers

Err

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.