Overfitted high-dimensional matrix factorizations via adaptive spectral shrinkage

• URL: http://arxiv.org/abs/2606.19540v1
• Date: Wed, 17 Jun 2026 19:33:03 GMT
• Title: Overfitted high-dimensional matrix factorizations via adaptive spectral shrinkage
• Abstract summary: We show that textttEigenBayes adapts to the signal-to-noise ratio of each outcome and latent dimension, while shrinking latent components to zero.<n>We demonstrate strong empirical performance in numerical experiments and a genomics application.
• Score: 8.549941732144035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Factor models are popular approaches for analyzing high-dimensional data to extract low-rank signals and estimate covariances. They decompose the covariance matrix as the sum of low-rank and diagonal components. A key issue is how to choose the latent dimension $k$, which is particularly challenging when the factor model only holds approximately and in low signal-to-noise scenarios. Bayesian overfitted factor models specify an upper bound on $k$ and rely on structured shrinkage priors to effectively remove extra components. Such approaches are popular and effective, but computationally expensive. We propose a much faster \texttt{EigenBayes} approach that provides valid uncertainty quantification, based on spectral estimation of latent factors and adaptive empirical Bayes calibration of key hyperparameters. The resulting posterior distribution factorizes across outcomes and is analytically tractable, bypassing Markov chain Monte Carlo. We show that \texttt{EigenBayes} adapts to the signal-to-noise ratio of each outcome and latent dimension, while shrinking superfluous latent components to zero. We establish favorable asymptotic properties and demonstrate strong empirical performance in numerical experiments and a genomics application, where EigenBayes outperforms state-of-the-art alternatives.

Related papers

• Inference on covariance structure in high-dimensional multi-view data [8.549941732144035]
  Posterior computation is conducted via expensive and brittle Markov chain Monte Carlo sampling or variational approximations.<n>Our proposed methodology employs spectral decompositions to estimate and align latent factors that are active in at least one view.<n>We show excellent performance in simulations, including accurate uncertainty, and apply the methodology to integrate four high-dimensional views from a multi-omics dataset of cancer cell samples.
  arXiv  Detail & Related papers  (2025-09-02T19:20:42Z)
• T-Rex: Fitting a Robust Factor Model via Expectation-Maximization [0.0]
  We propose a novel expectation-maximization (EM) algorithm for robustly fitting statistical factor models.<n>Our approach is based on Tyler's M-estimator of the scatter matrix for an elliptical distribution.<n>We present numerical experiments on both synthetic and real examples.
  arXiv  Detail & Related papers  (2025-05-17T18:53:06Z)
• Modeling High-Dimensional Dependent Data in the Presence of Many Explanatory Variables and Weak Signals [0.0]
  This article considers a novel and widely applicable approach to modeling high-dimensional dependent data when a large number of explanatory variables are available and the signal-to-noise ratio is low.
  arXiv  Detail & Related papers  (2024-12-06T02:54:31Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  We present a unifying perspective on recent results on ridge regression.<n>We use the basic tools of random matrix theory and free probability, aimed at readers with backgrounds in physics and deep learning.<n>Our results extend and provide a unifying perspective on earlier models of scaling laws.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• Fundamental limits of Non-Linear Low-Rank Matrix Estimation [18.455890316339595]
  Bayes-optimal performances are characterized by an equivalent Gaussian model with an effective prior. We show that to reconstruct the signal accurately, one requires a signal-to-noise ratio growing as $Nfrac 12 (1-1/k_F)$, where $k_F$ is the first non-zero Fisher information coefficient of the function.
  arXiv  Detail & Related papers  (2024-03-07T05:26:52Z)
• Optimal Differentially Private PCA and Estimation for Spiked Covariance Matrices [10.377683220196873]
  Estimating a covariance matrix and its associated principal components is a fundamental problem in contemporary statistics. We study optimal differentially private Principal Component Analysis (PCA) and covariance estimation within the spiked covariance model. We propose computationally efficient differentially private estimators and prove their minimax optimality for sub-Gaussian distributions.
  arXiv  Detail & Related papers  (2024-01-08T11:18:14Z)
• Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent [43.097493761380186]
  gradient algorithms are an efficient method of approximately solving linear systems. We show that gradient descent produces accurate predictions, even in cases where it does not converge quickly to the optimum. Experimentally, gradient descent achieves state-of-the-art performance on sufficiently large-scale or ill-conditioned regression tasks.
  arXiv  Detail & Related papers  (2023-06-20T15:07:37Z)
• Learning Unnormalized Statistical Models via Compositional Optimization [73.30514599338407]
  Noise-contrastive estimation(NCE) has been proposed by formulating the objective as the logistic loss of the real data and the artificial noise. In this paper, we study it a direct approach for optimizing the negative log-likelihood of unnormalized models.
  arXiv  Detail & Related papers  (2023-06-13T01:18:16Z)
• Efficient CDF Approximations for Normalizing Flows [64.60846767084877]
  We build upon the diffeomorphic properties of normalizing flows to estimate the cumulative distribution function (CDF) over a closed region. Our experiments on popular flow architectures and UCI datasets show a marked improvement in sample efficiency as compared to traditional estimators.
  arXiv  Detail & Related papers  (2022-02-23T06:11:49Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Slice Sampling for General Completely Random Measures [74.24975039689893]
  We present a novel Markov chain Monte Carlo algorithm for posterior inference that adaptively sets the truncation level using auxiliary slice variables. The efficacy of the proposed algorithm is evaluated on several popular nonparametric models.
  arXiv  Detail & Related papers  (2020-06-24T17:53:53Z)
• Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions [41.7567932118769]
  Empirical Risk Minimization algorithms are widely used in a variety of estimation and prediction tasks. In this paper, we characterize for the first time the fundamental limits on the statistical accuracy of convex ERM for inference.
  arXiv  Detail & Related papers  (2020-06-16T04:27:38Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-$\ell_1$-Norm Interpolated Classifiers [3.167685495996986]
  This paper establishes a precise high-dimensional theory for boosting on separable data. Under a class of statistical models, we provide an exact analysis of the universality error of boosting. We also explicitly pin down the relation between the boosting test error and the optimal Bayes error.
  arXiv  Detail & Related papers  (2020-02-05T00:24:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.