Support Recovery with Stochastic Gates: Theory and Application for Linear Models

• URL: http://arxiv.org/abs/2110.15960v2
• Date: Mon, 1 Nov 2021 17:59:28 GMT
• Title: Support Recovery with Stochastic Gates: Theory and Application for Linear Models
• Authors: Soham Jana, Henry Li, Yutaro Yamada, Ofir Lindenbaum
• Abstract summary: We analyze the problem of simultaneous support recovery and estimation of the coefficient vector ($beta*$) in a linear model with independent and identically distributed Normal errors. Considering design we show that under reasonable conditions on dimension and sparsity of $beta*$ the STG based estimator converges to the true data generating coefficient vector and also detects its support set with high probability.
• Score: 9.644417971611908
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We analyze the problem of simultaneous support recovery and estimation of the coefficient vector ($\beta^*$) in a linear model with independent and identically distributed Normal errors. We apply the penalized least square estimator based on non-linear penalties of stochastic gates (STG) [YLNK20] to estimate the coefficients. Considering Gaussian design matrices we show that under reasonable conditions on dimension and sparsity of $\beta^*$ the STG based estimator converges to the true data generating coefficient vector and also detects its support set with high probability. We propose a new projection based algorithm for linear models setup to improve upon the existing STG estimator that was originally designed for general non-linear models. Our new procedure outperforms many classical estimators for support recovery in synthetic data analysis.

Related papers

• Linear-cost unbiased posterior estimates for crossed effects and matrix factorization models via couplings [0.0]
  We design and analyze unbiased Markov chain Monte Carlo schemes based on couplings of blocked Gibbs samplers (BGSs) Our methodology is designed for and applicable to high-dimensional BGS with conditionally independent blocks.
  arXiv  Detail & Related papers  (2024-10-11T16:05:01Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models. We provide an introduction and review of recent results on these topics, aimed at readers with backgrounds in physics and deep learning.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• Computational-Statistical Gaps in Gaussian Single-Index Models [77.1473134227844]
  Single-Index Models are high-dimensional regression problems with planted structure. We show that computationally efficient algorithms, both within the Statistical Query (SQ) and the Low-Degree Polynomial (LDP) framework, necessarily require $Omega(dkstar/2)$ samples.
  arXiv  Detail & Related papers  (2024-03-08T18:50:19Z)
• Sparse PCA with Oracle Property [115.72363972222622]
  We propose a family of estimators based on the semidefinite relaxation of sparse PCA with novel regularizations. We prove that, another estimator within the family achieves a sharper statistical rate of convergence than the standard semidefinite relaxation of sparse PCA.
  arXiv  Detail & Related papers  (2023-12-28T02:52:54Z)
• High-dimensional analysis of double descent for linear regression with random projections [0.0]
  We consider linear regression problems with a varying number of random projections, where we provably exhibit a double descent curve for a fixed prediction problem. We first consider the ridge regression estimator and re-interpret earlier results using classical notions from non-parametric statistics. We then compute equivalents of the generalization performance (in terms of bias and variance) of the minimum norm least-squares fit with random projections, providing simple expressions for the double descent phenomenon.
  arXiv  Detail & Related papers  (2023-03-02T15:58:09Z)
• Learning Graphical Factor Models with Riemannian Optimization [70.13748170371889]
  This paper proposes a flexible algorithmic framework for graph learning under low-rank structural constraints. The problem is expressed as penalized maximum likelihood estimation of an elliptical distribution. We leverage geometries of positive definite matrices and positive semi-definite matrices of fixed rank that are well suited to elliptical models.
  arXiv  Detail & Related papers  (2022-10-21T13:19:45Z)
• Robust Regularized Low-Rank Matrix Models for Regression and Classification [14.698622796774634]
  We propose a framework for matrix variate regression models based on a rank constraint, vector regularization (e.g., sparsity), and a general loss function. We show that the algorithm is guaranteed to converge, all accumulation points of the algorithm have estimation errors in the order of $O(sqrtn)$ally and substantially attaining the minimax rate.
  arXiv  Detail & Related papers  (2022-05-14T18:03:48Z)
• Tuned Regularized Estimators for Linear Regression via Covariance Fitting [17.46329281993348]
  We consider the problem of finding tuned regularized parameter estimators for linear models. We show that three known optimal linear estimators belong to a wider class of estimators. We show that the resulting class of estimators yields tuned versions of known regularized estimators.
  arXiv  Detail & Related papers  (2022-01-21T16:08:08Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Low-Rank Matrix Estimation From Rank-One Projections by Unlifted Convex Optimization [9.492903649862761]
  We study an estimator with a formulation convex for recovery of low-rank matrices from rank-one projections. We show that under both models the estimator succeeds, with high probability, if the number of measurements exceeds $r2 (d+d_$2) up.
  arXiv  Detail & Related papers  (2020-04-06T14:57:54Z)
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models [80.22609163316459]
  We introduce an unbiased estimator of the log marginal likelihood and its gradients for latent variable models based on randomized truncation of infinite series. We show that models trained using our estimator give better test-set likelihoods than a standard importance-sampling based approach for the same average computational cost.
  arXiv  Detail & Related papers  (2020-04-01T11:49:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.