Risk and cross validation in ridge regression with correlated samples

• URL: http://arxiv.org/abs/2408.04607v2
• Date: Sun, 11 Aug 2024 19:50:59 GMT
• Title: Risk and cross validation in ridge regression with correlated samples
• Authors: Alexander Atanasov, Jacob A. Zavatone-Veth, Cengiz Pehlevan,
• Abstract summary: We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We further extend our analysis to the case where the test point has non-trivial correlations with the training set, setting often encountered in time series forecasting. We validate our theory across a variety of high dimensional data.
• Score: 72.59731158970894
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent years have seen substantial advances in our understanding of high-dimensional ridge regression, but existing theories assume that training examples are independent. By leveraging recent techniques from random matrix theory and free probability, we provide sharp asymptotics for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We demonstrate that in this setting, the generalized cross validation estimator (GCV) fails to correctly predict the out-of-sample risk. However, in the case where the noise residuals have the same correlations as the data points, one can modify the GCV to yield an efficiently-computable unbiased estimator that concentrates in the high-dimensional limit, which we dub CorrGCV. We further extend our asymptotic analysis to the case where the test point has nontrivial correlations with the training set, a setting often encountered in time series forecasting. Assuming knowledge of the correlation structure of the time series, this again yields an extension of the GCV estimator, and sharply characterizes the degree to which such test points yield an overly optimistic prediction of long-time risk. We validate the predictions of our theory across a variety of high dimensional data.

Related papers

• Semiparametric conformal prediction [79.6147286161434]
  Risk-sensitive applications require well-calibrated prediction sets over multiple, potentially correlated target variables. We treat the scores as random vectors and aim to construct the prediction set accounting for their joint correlation structure. We report desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent [8.0225129190882]
  We analyze the statistical properties of generalized cross-validation (GCV) and leave-one-out cross-validation (LOOCV) applied to early-stopped descent gradient (GD) We prove that GCV is generically inconsistent as an estimator of the prediction risk of early-stopped GD, even for a well-specified linear model with isotropic features. Our theory requires only mild assumptions on the data distribution and does not require the underlying regression function to be linear.
  arXiv  Detail & Related papers  (2024-02-26T18:07:27Z)
• Corrected generalized cross-validation for finite ensembles of penalized estimators [5.165142221427927]
  Generalized cross-validation (GCV) is a widely-used method for estimating the squared out-of-sample prediction risk. We show that GCV is inconsistent for any finite ensemble of size greater than one.
  arXiv  Detail & Related papers  (2023-10-02T17:38:54Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors [10.857775300638831]
  We explore prediction risk as well as estimation risk under more general regression error assumptions. Our findings suggest that the benefits of over parameterization can extend to time series, panel and grouped data.
  arXiv  Detail & Related papers  (2023-05-22T10:04:20Z)
• Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression [14.493176427999028]
  We study the benign overfitting theory in the prediction of the conditional average treatment effect (CATE) with linear regression models. We show that the T-learner fails to achieve the consistency except the random assignment, while the IPW-learner converges the risk to zero if the propensity score is known.
  arXiv  Detail & Related papers  (2022-02-10T18:51:52Z)
• Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
  We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
  arXiv  Detail & Related papers  (2022-01-26T19:59:14Z)
• Optimal regularizations for data generation with probabilistic graphical models [0.0]
  Empirically, well-chosen regularization schemes dramatically improve the quality of the inferred models. We consider the particular case of L 2 and L 1 regularizations in the Maximum A Posteriori (MAP) inference of generative pairwise graphical models.
  arXiv  Detail & Related papers  (2021-12-02T14:45:16Z)
• Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning [57.88785630755165]
  Empirical risk minimization (ERM) is the workhorse of machine learning, but its model-agnostic guarantees can fail when we use adaptively collected data. We study a generic importance sampling weighted ERM algorithm for using adaptively collected data to minimize the average of a loss function over a hypothesis class. For policy learning, we provide rate-optimal regret guarantees that close an open gap in the existing literature whenever exploration decays to zero.
  arXiv  Detail & Related papers  (2021-06-03T09:50:13Z)
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited [68.8204255655161]
  We introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. Corpor is based on non-parametric isotonic regression and implemented via the Pool-adjacent-violators (PAV) algorithm.
  arXiv  Detail & Related papers  (2020-08-07T08:22:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.