Cross-Audit Projection for Model Risk Prediction

• URL: http://arxiv.org/abs/2607.02328v1
• Date: Thu, 02 Jul 2026 15:35:18 GMT
• Title: Cross-Audit Projection for Model Risk Prediction
• Abstract summary: For binary classification via empirical risk, $K$-fold CV may perform poorly in estimating class-specific risks, even worse than the empirical estimator.<n>These findings motivate a novel two-step procedure for model risk prediction, termed cross-audit projection (CAP)<n>The resulting CAP estimator is first-orderally equivalent to the empirical risk while achieving second-order unbiasedness.
• Score: 0.12183405753834557
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For training-data-based model risk prediction, $K$-fold cross-validation~(CV) is widely used to mitigate the well-known over-optimism of the empirical risk and is often regarded as reliable. However, for binary classification via empirical risk minimization, our numerical studies reveal a surprising phenomenon: $K$-fold CV may perform poorly in estimating class-specific risks, even worse than the empirical estimator. We perform a higher-order asymptotic analysis showing that $K$-fold CV may converge at a slower rate, whereas the empirical estimator exhibits a second-order asymptotic bias that explains its over-optimism. These findings motivate a novel two-step procedure for model risk prediction, termed cross-audit projection (CAP). The cross-audit step adopts the same resampling scheme as $K$-fold CV to estimate over-optimism in subsamples, while the asymptotic-theory-informed projection step adjusts for the reduced sample size in bias correction of the empirical risk. The resulting CAP estimator is first-order asymptotically equivalent to the empirical risk while achieving second-order asymptotic unbiasedness. An accompanying inference procedure is also developed. Simulation studies support theoretical advantages of CAP and demonstrate favorable finite-sample performance. An application to breast cancer detection further illustrates the proposed method.

Related papers

• Gradient boosting for extremes: sampling theory and application to insurance [0.0]
  We develop a theory for gradient boosting applied to the estimation of co-dependent Generalized Peaks of distributions.<n>We show that it significantly reduces gradient correlation during training and improves convergence stability.<n>The gradient boosting approach yields a good fit for the tail of settlement cost distributions and reveals that the number of days to settlement is the dominant predictor of tail heaviness.
  arXiv  Detail & Related papers  (2026-06-12T08:51:00Z)
• From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators [0.3058685580689604]
  We derive the risk function of regularized empirical risk minimization estimators tuned by $n$-fold cross-validation (CV)<n>The out-of-sample prediction loss converges in distribution to the squared-error loss (risk function) of shrinkage estimators tuned by Stein's unbiased risk estimate (SURE)
  arXiv  Detail & Related papers  (2026-03-20T18:05:39Z)
• Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling [50.872910438715486]
  Large Language Models (LLMs) are typically evaluated for safety under single-shot or low-budget adversarial prompting.<n>We propose a scaling-aware Best-of-N estimation of risk, SABER, for modeling jailbreak vulnerability under Best-of-N sampling.
  arXiv  Detail & Related papers  (2026-01-30T06:54:35Z)
• Prediction-Powered Causal Inferences [59.98498488132307]
  We focus on Prediction-Powered Causal Inferences (PPCI)<n>We first show that conditional calibration guarantees valid PPCI at population level.<n>We then introduce a sufficient representation constraint transferring validity across experiments.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Mitigating optimistic bias in entropic risk estimation and optimization with an application to insurance [5.407319151576265]
  The entropic risk measure is widely used to account for tail risks associated with an uncertain loss.<n>To mitigate the bias in the empirical entropic risk estimator, we propose a strongly consistent bootstrapping procedure.<n>We show that our methods suggest a higher (and more accurate) premium to homeowners.
  arXiv  Detail & Related papers  (2024-09-30T04:02:52Z)
• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations.<n>We demonstrate that in this setting, the generalized cross validation estimator (GCV) fails to correctly predict the out-of-sample risk.<n>We further extend our analysis to the case where the test point has nontrivial correlations with the training set, a setting often encountered in time series forecasting.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Mitigating multiple descents: A model-agnostic framework for risk monotonization [84.6382406922369]
  We develop a general framework for risk monotonization based on cross-validation. We propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting.
  arXiv  Detail & Related papers  (2022-05-25T17:41:40Z)
• Surrogate Assisted Semi-supervised Inference for High Dimensional Risk Prediction [3.10560974227074]
  We develop a surrogate assisted semi-supervised-learning (SAS) approach to risk modeling with high dimensional predictors. We demonstrate that the SAS procedure provides valid inference for the predicted risk derived from a high dimensional working model.
  arXiv  Detail & Related papers  (2021-05-04T03:08:51Z)
• PAC$^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime [75.19403612525811]
  This work develops a multi-sample loss which can close the gap by spanning a trade-off between the two risks. Empirical study demonstrates improvement to the predictive distribution.
  arXiv  Detail & Related papers  (2020-10-19T16:08:34Z)
• A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
  A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution. We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
  arXiv  Detail & Related papers  (2020-07-08T11:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.