From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators

• URL: http://arxiv.org/abs/2603.20388v1
• Date: Fri, 20 Mar 2026 18:05:39 GMT
• Title: From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators
• Abstract summary: We derive the risk function of regularized empirical risk minimization estimators tuned by $n$-fold cross-validation (CV)<n>The out-of-sample prediction loss converges in distribution to the squared-error loss (risk function) of shrinkage estimators tuned by Stein's unbiased risk estimate (SURE)
• Score: 0.3058685580689604
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We derive the asymptotic risk function of regularized empirical risk minimization (ERM) estimators tuned by $n$-fold cross-validation (CV). The out-of-sample prediction loss of such estimators converges in distribution to the squared-error loss (risk function) of shrinkage estimators in the normal means model, tuned by Stein's unbiased risk estimate (SURE). This risk function provides a more fine-grained picture of predictive performance than uniform bounds on worst-case regret, which are common in learning theory: it quantifies how risk varies with the true parameter. As key intermediate steps, we show that (i) $n$-fold CV converges uniformly to SURE, and (ii) while SURE typically has multiple local minima, its global minimum is generically well separated. Well-separation ensures that uniform convergence of CV to SURE translates into convergence of the tuning parameter chosen by CV to that chosen by SURE.

Related papers

• Beyond Consistency: Inference for the Relative risk functional in Deep Nonparametric Cox Models [7.2275111013288535]
  It is unclear how Cox-based optimization error propagates to population risk under partial likelihood.<n>It is also unclear how pointwise bias can be controlled, and how ensemble-based uncertainty behaves under realistic variance regimes.<n>We develop an distribution theory for deep Cox estimators that addresses these issues.
  arXiv  Detail & Related papers  (2026-03-25T01:43:12Z)
• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations.<n>We demonstrate that in this setting, the generalized cross validation estimator (GCV) fails to correctly predict the out-of-sample risk.<n>We further extend our analysis to the case where the test point has nontrivial correlations with the training set, a setting often encountered in time series forecasting.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning [5.293069542318491]
  We employ random matrix theory to establish consistency of generalized cross validation (GCV) for estimating prediction risks of sketched ridge regression ensembles. For squared prediction risk, we provide a decomposition into an unsketched equivalent implicit ridge bias and a sketching-based variance, and prove that the risk can be globally tuning by only sketch size in infinite ensembles. We also propose an "ensemble trick" whereby the risk for unsketched ridge regression can be efficiently estimated via GCV using small sketched ridge ensembles.
  arXiv  Detail & Related papers  (2023-10-06T16:27:43Z)
• Corrected generalized cross-validation for finite ensembles of penalized estimators [5.165142221427927]
  Generalized cross-validation (GCV) is a widely-used method for estimating the squared out-of-sample prediction risk. We show that GCV is inconsistent for any finite ensemble of size greater than one.
  arXiv  Detail & Related papers  (2023-10-02T17:38:54Z)
• On the Variance, Admissibility, and Stability of Empirical Risk Minimization [57.63331017830154]
  Empirical Risk Minimization (ERM) may attain minimax suboptimal rates in terms of the mean squared error.<n>We prove that under relatively mild assumptions, the suboptimality of ERM must be due to its large bias.
  arXiv  Detail & Related papers  (2023-05-29T15:25:48Z)
• Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation [4.87717454493713]
  We study subsampling-based ridge ensembles in the proportionals regime. We prove that the risk of the optimal full ridgeless ensemble (fitted on all possible subsamples) matches that of the optimal ridge predictor.
  arXiv  Detail & Related papers  (2023-04-25T17:43:27Z)
• Extrapolated cross-validation for randomized ensembles [2.3609229325947885]
  This paper introduces a cross-validation method, ECV, for tuning the ensemble and subsample sizes in randomized ensembles. We show that ECV yields $delta$-optimal ensembles for squared prediction risk. In comparison to sample-split cross-validation and $K$-fold cross-validation, ECV achieves higher accuracy avoiding sample splitting.
  arXiv  Detail & Related papers  (2023-02-27T04:19:18Z)
• Mitigating multiple descents: A model-agnostic framework for risk monotonization [84.6382406922369]
  We develop a general framework for risk monotonization based on cross-validation. We propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting.
  arXiv  Detail & Related papers  (2022-05-25T17:41:40Z)
• Optimal variance-reduced stochastic approximation in Banach spaces [114.8734960258221]
  We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. We establish non-asymptotic bounds for both the operator defect and the estimation error.
  arXiv  Detail & Related papers  (2022-01-21T02:46:57Z)
• A Unifying Theory of Thompson Sampling for Continuous Risk-Averse Bandits [91.3755431537592]
  This paper unifies the analysis of risk-averse Thompson sampling algorithms for the multi-armed bandit problem. Using the contraction principle in the theory of large deviations, we prove novel concentration bounds for continuous risk functionals. We show that a wide class of risk functionals as well as "nice" functions of them satisfy the continuity condition.
  arXiv  Detail & Related papers  (2021-08-25T17:09:01Z)
• Near-optimal inference in adaptive linear regression [60.08422051718195]
  Even simple methods like least squares can exhibit non-normal behavior when data is collected in an adaptive manner. We propose a family of online debiasing estimators to correct these distributional anomalies in at least squares estimation. We demonstrate the usefulness of our theory via applications to multi-armed bandit, autoregressive time series estimation, and active learning with exploration.
  arXiv  Detail & Related papers  (2021-07-05T21:05:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.