Optimal ridge regularization revisited

• URL: http://arxiv.org/abs/2605.28679v1
• Date: Wed, 27 May 2026 16:12:11 GMT
• Title: Optimal ridge regularization revisited
• Abstract summary: We compute the optimal regularization strength numerically from the generative parameters in the fixed-$X$ setting.<n>Our experimental evaluation over synthetic data shows that the proposed procedure attains near-optimal random-$X$ generalization across a wide range of sample sizes, aspect ratios, and noise levels.
• Score: 3.437656066916039
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider $L^2$-regularized linear (ridge) regression over a finite data sample $X$ with bounded covariance and linear prediction targets $y$ with additive isotropic noise of finite variance. We present an iterative procedure to compute the optimal regularization strength numerically from the generative parameters in the fixed-$X$ setting and prove its convergence at limited noise levels. Our experimental evaluation over synthetic data shows that the proposed procedure combined with sample-based parameter estimates attains near-optimal random-$X$ generalization across a wide range of sample sizes, aspect ratios, and noise levels, at an added computational cost equivalent to one preliminary ridge regression in the underparameterized regime and two in the overparameterized case.

Related papers

• Causal Discovery in Equal Variance Linear Gaussian DAGs via SURE-Tuned Ridge Regression [18.884190660925444]
  SURE-Ridge is a non-iterative, closed-form estimator for equal variance linear Gaussian SEM.<n>It achieves the lowest structural Hamming distance in the small-sample regime and the lowest run time across all sample sizes tested.
  arXiv  Detail & Related papers  (2026-08-17T21:07:06Z)
• A Simple Approximation to the Distribution of the Ridge Regression Estimator [1.1678783855674757]
  In finite samples, the ridge regression estimator trades off bias and variance to reduce estimation and prediction error.<n>Our approximation is based on nonstandards where $i)$ we let the estimator's regularization parameter grow proportionally to the sample size.<n>We propose two new strategies to select the regularization parameter for the ridge regression estimator.
  arXiv  Detail & Related papers  (2026-08-03T17:28:17Z)
• Neural Optimal Transport Meets Multivariate Conformal Prediction [58.43397908730771]
  We propose a framework for conditional vectorile regression (CVQR)<n>CVQR combines neural optimal transport with quantized optimization, and apply it to predictions.
  arXiv  Detail & Related papers  (2025-09-29T19:50:19Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate.<n>We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Retire: Robust Expectile Regression in High Dimensions [3.9391041278203978]
  Penalized quantile and expectile regression methods offer useful tools to detect heteroscedasticity in high-dimensional data. We propose and study (penalized) robust expectile regression (retire) We show that the proposed procedure can be efficiently solved by a semismooth Newton coordinate descent algorithm.
  arXiv  Detail & Related papers  (2022-12-11T18:03:12Z)
• Distributed Estimation and Inference for Semi-parametric Binary Response Models [8.309294338998539]
  This paper studies the maximum score estimator of a semi-parametric binary choice model under a distributed computing environment. An intuitive divide-and-conquer estimator is computationally expensive and restricted by a non-regular constraint on the number of machines.
  arXiv  Detail & Related papers  (2022-10-15T23:06:46Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• Tuning Stochastic Gradient Algorithms for Statistical Inference via Large-Sample Asymptotics [18.93569692490218]
  tuning of gradient algorithms often based on trial-and-error rather than generalizable theory. We show that averaging with a large fixed step size is robust to the choice of tuning parameters. We lay the foundation for a systematic analysis of other gradient Monte Carlo algorithms.
  arXiv  Detail & Related papers  (2022-07-25T17:58:09Z)
• Noise Covariance Estimation in Multi-Task High-dimensional Linear Models [8.807375890824977]
  This paper studies the multi-task high-dimensional linear regression models where the noise among different tasks is correlated. Treating the regression coefficients as a nuisance parameter, we leverage the multi-task elastic-net and multi-task lasso estimators to estimate the nuisance. Under suitable conditions, the proposed estimator attains the same rate of convergence as the "oracle" estimator.
  arXiv  Detail & Related papers  (2022-06-15T02:37:37Z)
• Heavy-tailed Streaming Statistical Estimation [58.70341336199497]
  We consider the task of heavy-tailed statistical estimation given streaming $p$ samples. We design a clipped gradient descent and provide an improved analysis under a more nuanced condition on the noise of gradients.
  arXiv  Detail & Related papers  (2021-08-25T21:30:27Z)
• On the Convergence of Stochastic Extragradient for Bilinear Games with Restarted Iteration Averaging [96.13485146617322]
  We present an analysis of the ExtraGradient (SEG) method with constant step size, and present variations of the method that yield favorable convergence. We prove that when augmented with averaging, SEG provably converges to the Nash equilibrium, and such a rate is provably accelerated by incorporating a scheduled restarting procedure.
  arXiv  Detail & Related papers  (2021-06-30T17:51:36Z)
• Support recovery and sup-norm convergence rates for sparse pivotal estimation [79.13844065776928]
  In high dimensional sparse regression, pivotal estimators are estimators for which the optimal regularization parameter is independent of the noise level. We show minimax sup-norm convergence rates for non smoothed and smoothed, single task and multitask square-root Lasso-type estimators.
  arXiv  Detail & Related papers  (2020-01-15T16:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.