High-Dimensional Interpolators Can Be Fragile: Heavy Tails and High-Dimensional Large Deviations

• URL: http://arxiv.org/abs/2607.09547v1
• Date: Fri, 10 Jul 2026 15:58:03 GMT
• Title: High-Dimensional Interpolators Can Be Fragile: Heavy Tails and High-Dimensional Large Deviations
• Abstract summary: We study high-dimensional linear interpolators using large-deviation methods.<n>We first show that the risk of ridgeless regression can exhibit heavy-tailed behavior.<n>We quantify this phenomenon at the level of large-deviation rates.
• Score: 3.540245474029962
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-dimensional interpolation is common in modern machine learning, but its tail risk is less understood than its expected prediction risk. Existing theory shows that interpolating models can perform well in expectation, yet such guarantees do not determine the probability of rare, severe errors. In operations research and stochastic decision-making applications, rare estimation errors can have disproportionate downstream effects, so tail behavior matters alongside average performance. We study the fragility of high-dimensional linear interpolators using large-deviation methods. We focus on ridgeless regression and compare it with ridge-regularized estimators. We first show that the risk of ridgeless regression can exhibit heavy-tailed behavior: although its expected risk may remain well controlled, its upper tail can decay much more slowly than that of regularized alternatives. We then quantify this phenomenon at the level of large-deviation rates. In the regime we study, ridge regularization suppresses fixed right-tail deviations at the $n^2$ scale, whereas ridgeless regression has only $n\log n$-scale decay, where $n$ is the sample size. This gap shows that interpolation can be statistically fragile even when it is accurate on average. Thus regularization affects the frequency of rare, high-impact risk events in addition to the usual bias-variance tradeoff.

Related papers

• Double descent for least-squares interpolation on contaminated data: A simulation study [0.0]
  "Double descent" indicates that the generalization error decreases after a certain complexity model has been reached.<n>We compare the performance of the highly non-robust least-squares estimator with several robust alternatives.<n>It turns out that large overparametrization indeed allows for a double descent phenomenon, resulting in a very good generalization performance of the least-squares interpolator.
  arXiv  Detail & Related papers  (2026-04-15T16:31:23Z)
• Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling [50.872910438715486]
  Large Language Models (LLMs) are typically evaluated for safety under single-shot or low-budget adversarial prompting.<n>We propose a scaling-aware Best-of-N estimation of risk, SABER, for modeling jailbreak vulnerability under Best-of-N sampling.
  arXiv  Detail & Related papers  (2026-01-30T06:54:35Z)
• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations.<n>We demonstrate that in this setting, the generalized cross validation estimator (GCV) fails to correctly predict the out-of-sample risk.<n>We further extend our analysis to the case where the test point has nontrivial correlations with the training set, a setting often encountered in time series forecasting.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• Robust deep learning from weakly dependent data [0.0]
  This paper considers robust deep learning from weakly dependent observations, with unbounded loss function and unbounded input/output. We derive a relationship between these bounds and $r$, and when the data have moments of any order (that is $r=infty$), the convergence rate is close to some well-known results.
  arXiv  Detail & Related papers  (2024-05-08T14:25:40Z)
• A Stable and Efficient Covariate-Balancing Estimator for Causal Survival Effects [33.62020029799266]
  We address the problem of estimating survival causal effects in data with conditionally-independent censoring. This addresses the use of inverses of small estimated probabilities and the resulting amplification of estimation error. We validate our theoretical results in experiments on synthetic and semi-synthetic data.
  arXiv  Detail & Related papers  (2023-10-01T00:04:29Z)
• Quantifying predictive uncertainty of aphasia severity in stroke patients with sparse heteroscedastic Bayesian high-dimensional regression [47.1405366895538]
  Sparse linear regression methods for high-dimensional data commonly assume that residuals have constant variance, which can be violated in practice. This paper proposes estimating high-dimensional heteroscedastic linear regression models using a heteroscedastic partitioned empirical Bayes Expectation Conditional Maximization algorithm.
  arXiv  Detail & Related papers  (2023-09-15T22:06:29Z)
• Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors [10.857775300638831]
  We explore prediction risk as well as estimation risk under more general regression error assumptions. Our findings suggest that the benefits of over parameterization can extend to time series, panel and grouped data.
  arXiv  Detail & Related papers  (2023-05-22T10:04:20Z)
• On Regression in Extreme Regions [0.7974430263940756]
  We establish a statistical learning theoretical framework aimed at extrapolation, or out-of-domain generalization, on the unobserved tails of covariates.<n>We address the stylized problem of nonparametric least squares regression with predictors chosen from a Vapnik-Chervonenkis class.<n>We quantify the predictive performance on tail regions in terms of excess risk, presenting it as a finite sample risk bound with a clear bias-variance decomposition.
  arXiv  Detail & Related papers  (2023-03-06T12:55:38Z)
• The Implicit Bias of Benign Overfitting [31.714928102950584]
  benign overfitting is where a predictor perfectly fits noisy training data while attaining near-optimal expected loss. We show how this can be extended beyond standard linear regression. We then turn to classification problems, and show that the situation there is much more favorable.
  arXiv  Detail & Related papers  (2022-01-27T12:49:21Z)
• Interpolation can hurt robust generalization even when there is no noise [76.3492338989419]
  We show that avoiding generalization through ridge regularization can significantly improve generalization even in the absence of noise. We prove this phenomenon for the robust risk of both linear regression and classification and hence provide the first theoretical result on robust overfitting.
  arXiv  Detail & Related papers  (2021-08-05T23:04:15Z)
• Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression [91.3373131262391]
  Uncertainty is the only certainty there is. Traditionally, the direct regression formulation is considered and the uncertainty is modeled by modifying the output space to a certain family of probabilistic distributions. How to model the uncertainty within the present-day technologies for regression remains an open issue.
  arXiv  Detail & Related papers  (2021-03-25T06:56:09Z)
• SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression [68.66245730450915]
  We develop an improved method for debiasing predictions and estimating frequentist uncertainty for practical datasets. Our main contribution is SLOE, an estimator of the signal strength with convergence guarantees that reduces the computation time of estimation and inference by orders of magnitude.
  arXiv  Detail & Related papers  (2021-03-23T17:48:56Z)
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy [88.51943635427709]
  Adversarial training augments the training set with perturbations to improve the robust error. We show that the standard error could increase even when the augmented perturbations have noiseless observations from the optimal linear predictor.
  arXiv  Detail & Related papers  (2020-02-25T08:03:01Z)
• Interpolating Predictors in High-Dimensional Factor Regression [2.1055643409860743]
  This work studies finite-sample properties of the risk of the minimum-norm interpolating predictor in high-dimensional regression models. We show that the min-norm interpolating predictor can have similar risk to predictors based on principal components regression and ridge regression, and can improve over LASSO based predictors, in the high-dimensional regime.
  arXiv  Detail & Related papers  (2020-02-06T22:08:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.