Calibrated Principal Component Regression

• URL: http://arxiv.org/abs/2510.19020v1
• Date: Tue, 21 Oct 2025 19:01:42 GMT
• Title: Calibrated Principal Component Regression
• Abstract summary: Principal Component Regression (PCR) incurs truncation bias when the true regression vector has mass outside the retained principal components (PC)<n>We propose Calibrated Principal Component Regression (CPCR), which first learns a low-variance prior in the PC subspace and then calibrates the model in the original feature space via a centered Tikhonov step.
• Score: 5.890015449615021
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new method for statistical inference in generalized linear models. In the overparameterized regime, Principal Component Regression (PCR) reduces variance by projecting high-dimensional data to a low-dimensional principal subspace before fitting. However, PCR incurs truncation bias whenever the true regression vector has mass outside the retained principal components (PC). To mitigate the bias, we propose Calibrated Principal Component Regression (CPCR), which first learns a low-variance prior in the PC subspace and then calibrates the model in the original feature space via a centered Tikhonov step. CPCR leverages cross-fitting and controls the truncation bias by softening PCR's hard cutoff. Theoretically, we calculate the out-of-sample risk in the random matrix regime, which shows that CPCR outperforms standard PCR when the regression signal has non-negligible components in low-variance directions. Empirically, CPCR consistently improves prediction across multiple overparameterized problems. The results highlight CPCR's stability and flexibility in modern overparameterized settings.

Related papers

• De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction [11.327284468812948]
  Principal component regression regularizes high-dimensional prediction by choosing a spectral cutoff.<n>We prove an ordinary-PCR prediction-risk lower bound uniform over all ranks and a high-probability dPCR upper bound.<n>An exact risk decomposition explains the separation: denominator inflation is governed by first spectral mass, whereas the clean prediction cost of correction is governed by squared spectral mass.
  arXiv  Detail & Related papers  (2026-07-18T04:50:59Z)
• Perturb and Correct: Post-Hoc Ensembles using Affine Redundancy [0.0]
  We introduce Perturb-and-Correct (P&C), a method for constructing diverse predictors from a single pretrained network.<n>P&C applies random hidden layer perturbations with a least-squares correction in the subsequent affine layer, producing predictors that agree on calibration data while remaining free to disagree away from it.<n> Empirically, P&C achieves a strong ID/OOD tradeoff across MuJoCo dynamics prediction and CIFAR-10 OOD detection, matching or outperforming standard post-hoc baselines while requiring only a single pretrained model.
  arXiv  Detail & Related papers  (2026-05-02T22:48:11Z)
• Analysis of Nystrom method with sequential ridge leverage scores [69.32538992633842]
  Large-scale kernel ridge regression (KRR) is limited by the need to store a large kernel matrix K_t.<n>Recent works show that a sampling distribution proportional to the ridge leverage scores (RLSs) provides strong reconstruction guarantees for the approximation.<n>We introduce the INK-ESTIMATE algorithm, that incrementally computes the RLSs estimates.
  arXiv  Detail & Related papers  (2026-04-22T00:49:25Z)
• Predictive Spectral Calibration for Source-Free Test-Time Regression [0.764671395172401]
  Predictive Spectral (PSC) is a source-free framework that extends subspace alignment to block spectral matching.<n> PSC remains simple to implement, model-agnostic, and compatible with off-the-shelf pretrained regressors.
  arXiv  Detail & Related papers  (2026-03-10T08:15:44Z)
• One Sample is Enough to Make Conformal Prediction Robust [53.78604391939934]
  We show that conformal prediction attains some robustness even with a forward pass on a single randomly perturbed input.<n>Our approach returns robust sets with smaller average set size compared to SOTA methods which use many (e.g. around 100) passes per input.
  arXiv  Detail & Related papers  (2025-06-19T19:14:25Z)
• A Simplified Analysis of SGD for Linear Regression with Weight Averaging [64.2393952273612]
  Recent work bycitetzou 2021benign provides sharp rates for SGD optimization in linear regression using constant learning rate.<n>We provide a simplified analysis recovering the same bias and variance bounds provided incitepzou 2021benign based on simple linear algebra tools.<n>We believe our work makes the analysis of gradient descent on linear regression very accessible and will be helpful in further analyzing mini-batching and learning rate scheduling.
  arXiv  Detail & Related papers  (2025-06-18T15:10:38Z)
• Adaptive Multi-step Refinement Network for Robust Point Cloud Registration [82.64560249066734]
  Point Cloud Registration estimates the relative rigid transformation between two point clouds of the same scene.<n>We propose an adaptive multi-step refinement network that refines the registration quality at each step by leveraging the information from the preceding step.<n>Our method achieves state-of-the-art performance on both the 3DMatch/3DLoMatch and KITTI benchmarks.
  arXiv  Detail & Related papers  (2023-12-05T18:59:41Z)
• Classifier Calibration with ROC-Regularized Isotonic Regression [0.0]
  We use isotonic regression to minimize the cross entropy on a calibration set via monotone transformations. IR acts as an adaptive binning procedure, which allows achieving a calibration error of zero, but leaves open the issue of the effect on performance. We show empirically that this general monotony criterion is effective in striking a balance between reducing cross entropy loss and avoiding overfitting of the calibration set.
  arXiv  Detail & Related papers  (2023-11-21T08:45:09Z)
• Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms [88.74308282658133]
  Reization (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. Recent studies have revealed that, when applied to long-term reinforcement learning problems, model-based RP PGMs may experience chaotic and non-smooth optimization landscapes. We propose a spectral normalization method to mitigate the exploding variance issue caused by long model unrolls.
  arXiv  Detail & Related papers  (2023-10-30T18:43:21Z)
• Adaptive Principal Component Regression with Applications to Panel Data [29.295938927701396]
  We provide the first time-uniform finite sample guarantees for (regularized) Principal component regression. Our results rely on adapting tools from modern martingale concentration to the error-in-variables setting. We show that our method empirically outperforms a baseline which does not leverage error-in-variables regression.
  arXiv  Detail & Related papers  (2023-07-03T21:13:40Z)
• Dimensionality Reduction and Wasserstein Stability for Kernel Regression [1.3812010983144802]
  We study consequences of the naive two-step procedure where first the dimension of the input variables is reduced and second, the reduced input variables are used to predict the output variable with kernel regression. In order to analyze the resulting regression errors, a novel stability result for kernel regression with respect to the Wasserstein distance is derived.
  arXiv  Detail & Related papers  (2022-03-17T14:26:28Z)
• Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey [5.967999555890417]
  This is a tutorial and survey paper on various methods for Sufficient Dimension Reduction (SDR) We cover these methods with both statistical high-dimensional regression perspective and machine learning approach for dimensionality reduction.
  arXiv  Detail & Related papers  (2021-10-18T21:05:08Z)
• AgFlow: Fast Model Selection of Penalized PCA via Implicit Regularization Effects of Gradient Flow [64.81110234990888]
  Principal component analysis (PCA) has been widely used as an effective technique for feature extraction and dimension reduction. In the High Dimension Low Sample Size (HDLSS) setting, one may prefer modified principal components, with penalized loadings. We propose Approximated Gradient Flow (AgFlow) as a fast model selection method for penalized PCA.
  arXiv  Detail & Related papers  (2021-10-07T08:57:46Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• On Model Identification and Out-of-Sample Prediction of Principal Component Regression: Applications to Synthetic Controls [20.96904429337912]
  We analyze principal component regression (PCR) in a high-dimensional error-in-variables setting with fixed design. We establish non-asymptotic out-of-sample prediction guarantees that improve upon the best known rates.
  arXiv  Detail & Related papers  (2020-10-27T17:07:36Z)
• Sparse principal component regression via singular value decomposition approach [0.0]
  Principal component regression (PCR) is a two-stage procedure. Since PCA is performed by using only explanatory variables, the principal components have no information about the response variable. We propose a one-stage procedure forPCR in terms of singular value decomposition approach.
  arXiv  Detail & Related papers  (2020-02-21T09:03:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.