Evaluation and Optimization of Leave-one-out Cross-validation for the Lasso

• URL: http://arxiv.org/abs/2508.14368v2
• Date: Sat, 01 Nov 2025 03:55:05 GMT
• Title: Evaluation and Optimization of Leave-one-out Cross-validation for the Lasso
• Authors: Ryan Burn,
• Abstract summary: I develop an algorithm to produce the piecewise quadratic that computes leave-one-out cross-validation for the lasso.<n>I show how the algorithm can be modified to compute approximate leave-one-out cross-validation, making it suitable for larger data sets.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: I develop an algorithm to produce the piecewise quadratic that computes leave-one-out cross-validation for the lasso as a function of its hyperparameter. The algorithm can be used to find exact hyperparameters that optimize leave-one-out cross-validation either globally or locally, and its practicality is demonstrated on real-world data sets. I also show how the algorithm can be modified to compute approximate leave-one-out cross-validation, making it suitable for larger data sets.

Related papers

• Search-Based Correction of Reasoning Chains for Language Models [72.61861891295302]
  Chain-of-Thought (CoT) reasoning has advanced the capabilities and transparency of language models (LMs)<n>We introduce a new self-correction framework that augments each reasoning step in a CoT with a latent variable indicating its veracity.<n>We also introduce Search Corrector, a discrete search algorithm over-valued veracity assignments.
  arXiv  Detail & Related papers  (2025-05-17T04:16:36Z)
• Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models [2.4901555666568624]
  This study introduces NACHOS to reduce and quantify the variance of test performance metrics of deep learning models.<n> NACHOS integrates NCV and AHPO within a parallelized high-performance computing framework.<n>DACHOS is introduced to leverage AHPO and cross-validation to build the final model on the full dataset.
  arXiv  Detail & Related papers  (2025-03-11T16:25:44Z)
• Pathwise optimization for bridge-type estimators and its applications [49.1574468325115]
  Pathwise methods allow to efficiently compute the full path for penalized estimators.<n>We apply these algorithms to the penalized estimation of processes observed at discrete times.
  arXiv  Detail & Related papers  (2024-12-05T10:38:29Z)
• Optimal Cross-Validation for Sparse Linear Regression [5.156484100374059]
  We use k-fold cross-validation to select sparsity and robustness of linear regressors.<n>Cross-validation substantially increases the computational cost of sparse regression.<n>We improve upon this state of affairs by solving 50-80% fewer mixed-integer optimization problems.
  arXiv  Detail & Related papers  (2023-06-26T17:02:45Z)
• Provably Efficient Bayesian Optimization with Unknown Gaussian Process Hyperparameter Estimation [44.53678257757108]
  We propose a new BO method that can sub-linearly converge to the objective function's global optimum. Our method uses a multi-armed bandit technique (EXP3) to add random data points to the BO process. We demonstrate empirically that our method outperforms existing approaches on various synthetic and real-world problems.
  arXiv  Detail & Related papers  (2023-06-12T03:35:45Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations [76.82124752950148]
  We develop a convenient gradient-based method for selecting the data augmentation. We use a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as our objective.
  arXiv  Detail & Related papers  (2022-02-22T02:51:11Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Overfitting in Bayesian Optimization: an empirical study and early-stopping solution [41.782410830989136]
  We propose the first problem-adaptive and interpretable criterion to early stop BO. We show that our approach can substantially reduce compute time with little to no loss of test accuracy.
  arXiv  Detail & Related papers  (2021-04-16T15:26:23Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Fast calculation of Gaussian Process multiple-fold cross-validation residuals and their covariances [0.6091702876917281]
  We generalize fast leave-one-out formulae to multiple-fold cross-validation. We highlight the covariance structure of cross-validation residuals in both Simple and Universal Kriging frameworks. Our results enable fast multiple-fold cross-validation and have direct consequences in model diagnostics.
  arXiv  Detail & Related papers  (2021-01-08T17:02:37Z)
• Sparse PCA via $l_{2,p}$-Norm Regularization for Unsupervised Feature Selection [138.97647716793333]
  We propose a simple and efficient unsupervised feature selection method, by combining reconstruction error with $l_2,p$-norm regularization. We present an efficient optimization algorithm to solve the proposed unsupervised model, and analyse the convergence and computational complexity of the algorithm theoretically.
  arXiv  Detail & Related papers  (2020-12-29T04:08:38Z)
• Optimizing Approximate Leave-one-out Cross-validation to Tune Hyperparameters [0.0]
  We derive efficient formulas to compute the hessian gradient and the gradient of ALO. We demonstrate the usefulness of the proposed approach by finding hyper parameters for regularized logistic regression and ridge regression on various real-world data sets.
  arXiv  Detail & Related papers  (2020-11-20T04:57:41Z)
• Asymptotic Analysis of an Ensemble of Randomly Projected Linear Discriminants [94.46276668068327]
  In [1], an ensemble of randomly projected linear discriminants is used to classify datasets. We develop a consistent estimator of the misclassification probability as an alternative to the computationally-costly cross-validation estimator. We also demonstrate the use of our estimator for tuning the projection dimension on both real and synthetic data.
  arXiv  Detail & Related papers  (2020-04-17T12:47:04Z)
• Online Hyperparameter Search Interleaved with Proximal Parameter Updates [9.543667840503739]
  We develop a method that relies on the structure of proximal gradient methods and does not require a smooth cost function. Such a method is applied to Leave-one-out (LOO)-validated Lasso and Group Lasso. Numerical experiments corroborate the convergence of the proposed method to a local optimum of the LOO validation error curve.
  arXiv  Detail & Related papers  (2020-04-06T15:54:03Z)
• Towards new cross-validation-based estimators for Gaussian process regression: efficient adjoint computation of gradients [0.0]
  We suggest using new cross-validation criteria derived from the literature of scoring rules. We also provide an efficient method for computing the gradient of a cross-validation criterion.
  arXiv  Detail & Related papers  (2020-02-26T14:50:54Z)
• Implicit differentiation of Lasso-type models for hyperparameter optimization [82.73138686390514]
  We introduce an efficient implicit differentiation algorithm, without matrix inversion, tailored for Lasso-type problems. Our approach scales to high-dimensional data by leveraging the sparsity of the solutions.
  arXiv  Detail & Related papers  (2020-02-20T18:43:42Z)
• Support recovery and sup-norm convergence rates for sparse pivotal estimation [79.13844065776928]
  In high dimensional sparse regression, pivotal estimators are estimators for which the optimal regularization parameter is independent of the noise level. We show minimax sup-norm convergence rates for non smoothed and smoothed, single task and multitask square-root Lasso-type estimators.
  arXiv  Detail & Related papers  (2020-01-15T16:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.