A variational Bayes approach to debiased inference for low-dimensional parameters in high-dimensional linear regression

• URL: http://arxiv.org/abs/2406.12659v1
• Date: Tue, 18 Jun 2024 14:27:44 GMT
• Title: A variational Bayes approach to debiased inference for low-dimensional parameters in high-dimensional linear regression
• Authors: Ismaël Castillo, Alice L'Huillier, Kolyan Ray, Luke Travis,
• Abstract summary: We propose a scalable variational Bayes method for statistical inference in sparse linear regression. Our approach relies on assigning a mean-field approximation to the nuisance coordinates. This requires only a preprocessing step and preserves the computational advantages of mean-field variational Bayes.
• Score: 2.7498981662768536
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a scalable variational Bayes method for statistical inference for a single or low-dimensional subset of the coordinates of a high-dimensional parameter in sparse linear regression. Our approach relies on assigning a mean-field approximation to the nuisance coordinates and carefully modelling the conditional distribution of the target given the nuisance. This requires only a preprocessing step and preserves the computational advantages of mean-field variational Bayes, while ensuring accurate and reliable inference for the target parameter, including for uncertainty quantification. We investigate the numerical performance of our algorithm, showing that it performs competitively with existing methods. We further establish accompanying theoretical guarantees for estimation and uncertainty quantification in the form of a Bernstein--von Mises theorem.

Related papers

• Trust-Region Sequential Quadratic Programming for Stochastic Optimization with Random Models [57.52124921268249]
  We propose a Trust Sequential Quadratic Programming method to find both first and second-order stationary points. To converge to first-order stationary points, our method computes a gradient step in each iteration defined by minimizing a approximation of the objective subject. To converge to second-order stationary points, our method additionally computes an eigen step to explore the negative curvature the reduced Hessian matrix.
  arXiv  Detail & Related papers  (2024-09-24T04:39:47Z)
• Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models [0.0]
  We show how diffusion-based models can be repurposed for performing principled, identifiable Bayesian inference. We show how such maps can be learned via standard DBM training using a novel noise schedule. The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space.
  arXiv  Detail & Related papers  (2024-07-11T19:58:19Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate. We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Variational Bayesian surrogate modelling with application to robust design optimisation [0.9626666671366836]
  Surrogate models provide a quick-to-evaluate approximation to complex computational models. We consider Bayesian inference for constructing statistical surrogates with input uncertainties and dimensionality reduction. We demonstrate intrinsic and robust structural optimisation problems where cost functions depend on a weighted sum of the mean and standard deviation of model outputs.
  arXiv  Detail & Related papers  (2024-04-23T09:22:35Z)
• Finite Sample Confidence Regions for Linear Regression Parameters Using Arbitrary Predictors [1.6860963320038902]
  We explore a novel methodology for constructing confidence regions for parameters of linear models, using predictions from any arbitrary predictor. The derived confidence regions can be cast as constraints within a Mixed Linear Programming framework, enabling optimisation of linear objectives. Unlike previous methods, the confidence region can be empty, which can be used for hypothesis testing.
  arXiv  Detail & Related papers  (2024-01-27T00:15:48Z)
• Leveraging Self-Consistency for Data-Efficient Amortized Bayesian Inference [9.940560505044122]
  We propose a method to improve the efficiency and accuracy of amortized Bayesian inference. We estimate the marginal likelihood based on approximate representations of the joint model.
  arXiv  Detail & Related papers  (2023-10-06T17:41:41Z)
• The Implicit Delta Method [61.36121543728134]
  In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of uncertainty. We show that the change in the evaluation due to regularization is consistent for the variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference.
  arXiv  Detail & Related papers  (2022-11-11T19:34:17Z)
• CovarianceNet: Conditional Generative Model for Correct Covariance Prediction in Human Motion Prediction [71.31516599226606]
  We present a new method to correctly predict the uncertainty associated with the predicted distribution of future trajectories. Our approach, CovariaceNet, is based on a Conditional Generative Model with Gaussian latent variables.
  arXiv  Detail & Related papers  (2021-09-07T09:38:24Z)
• Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation [99.92568326314667]
  We propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principle. We demonstrate that ACNML compares favorably to a number of prior techniques for uncertainty estimation in terms of calibration on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-11-05T08:04:34Z)
• Support estimation in high-dimensional heteroscedastic mean regression [2.28438857884398]
  We consider a linear mean regression model with random design and potentially heteroscedastic, heavy-tailed errors. We use a strictly convex, smooth variant of the Huber loss function with tuning parameter depending on the parameters of the problem. For the resulting estimator we show sign-consistency and optimal rates of convergence in the $ell_infty$ norm.
  arXiv  Detail & Related papers  (2020-11-03T09:46:31Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.