Transfer Learning with Random Coefficient Ridge Regression

• URL: http://arxiv.org/abs/2306.15915v1
• Date: Wed, 28 Jun 2023 04:36:37 GMT
• Title: Transfer Learning with Random Coefficient Ridge Regression
• Authors: Hongzhe Zhang and Hongzhe Li
• Abstract summary: Ridge regression with random coefficients provides an important alternative to fixed coefficients regression in high dimensional setting. This paper considers estimation and prediction of random coefficient ridge regression in the setting of transfer learning.
• Score: 2.0813318162800707
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Ridge regression with random coefficients provides an important alternative to fixed coefficients regression in high dimensional setting when the effects are expected to be small but not zeros. This paper considers estimation and prediction of random coefficient ridge regression in the setting of transfer learning, where in addition to observations from the target model, source samples from different but possibly related regression models are available. The informativeness of the source model to the target model can be quantified by the correlation between the regression coefficients. This paper proposes two estimators of regression coefficients of the target model as the weighted sum of the ridge estimates of both target and source models, where the weights can be determined by minimizing the empirical estimation risk or prediction risk. Using random matrix theory, the limiting values of the optimal weights are derived under the setting when $p/n \rightarrow \gamma$, where $p$ is the number of the predictors and $n$ is the sample size, which leads to an explicit expression of the estimation or prediction risks. Simulations show that these limiting risks agree very well with the empirical risks. An application to predicting the polygenic risk scores for lipid traits shows such transfer learning methods lead to smaller prediction errors than the single sample ridge regression or Lasso-based transfer learning.

Related papers

• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We further extend our analysis to the case where the test point has non-trivial correlations with the training set, setting often encountered in time series forecasting. We validate our theory across a variety of high dimensional data.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate. We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• High-dimensional analysis of ridge regression for non-identically distributed data with a variance profile [0.0]
  We study the predictive risk of the ridge estimator for linear regression with a variance profile. For certain class of variance profile, our work highlights the emergence of the well-known double descent phenomenon. We also investigate the similarities and differences that exist with the standard setting of independent and identically distributed data.
  arXiv  Detail & Related papers  (2024-03-29T14:24:49Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression [14.493176427999028]
  We study the benign overfitting theory in the prediction of the conditional average treatment effect (CATE) with linear regression models. We show that the T-learner fails to achieve the consistency except the random assignment, while the IPW-learner converges the risk to zero if the propensity score is known.
  arXiv  Detail & Related papers  (2022-02-10T18:51:52Z)
• Understanding the Under-Coverage Bias in Uncertainty Estimation [58.03725169462616]
  quantile regression tends to emphunder-cover than the desired coverage level in reality. We prove that quantile regression suffers from an inherent under-coverage bias. Our theory reveals that this under-coverage bias stems from a certain high-dimensional parameter estimation error.
  arXiv  Detail & Related papers  (2021-06-10T06:11:55Z)
• Flexible Model Aggregation for Quantile Regression [92.63075261170302]
  Quantile regression is a fundamental problem in statistical learning motivated by a need to quantify uncertainty in predictions. We investigate methods for aggregating any number of conditional quantile models. All of the models we consider in this paper can be fit using modern deep learning toolkits.
  arXiv  Detail & Related papers  (2021-02-26T23:21:16Z)
• Dimensionality reduction, regularization, and generalization in overparameterized regressions [8.615625517708324]
  We show that PCA-OLS, also known as principal component regression, can be avoided with a dimensionality reduction. We show that dimensionality reduction improves robustness while OLS is arbitrarily susceptible to adversarial attacks. We find that methods in which the projection depends on the training data can outperform methods where the projections are chosen independently of the training data.
  arXiv  Detail & Related papers  (2020-11-23T15:38:50Z)
• Prediction in latent factor regression: Adaptive PCR and beyond [2.9439848714137447]
  We prove a master theorem that establishes a risk bound for a large class of predictors. We use our main theorem to recover known risk bounds for the minimum-norm interpolating predictor. We conclude with a detailed simulation study to support and complement our theoretical results.
  arXiv  Detail & Related papers  (2020-07-20T12:42:47Z)
• Interpolating Predictors in High-Dimensional Factor Regression [2.1055643409860743]
  This work studies finite-sample properties of the risk of the minimum-norm interpolating predictor in high-dimensional regression models. We show that the min-norm interpolating predictor can have similar risk to predictors based on principal components regression and ridge regression, and can improve over LASSO based predictors, in the high-dimensional regime.
  arXiv  Detail & Related papers  (2020-02-06T22:08:36Z)
• Censored Quantile Regression Forest [81.9098291337097]
  We develop a new estimating equation that adapts to censoring and leads to quantile score whenever the data do not exhibit censoring. The proposed procedure named it censored quantile regression forest, allows us to estimate quantiles of time-to-event without any parametric modeling assumption.
  arXiv  Detail & Related papers  (2020-01-08T23:20:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.