The Reciprocal Bayesian LASSO

• URL: http://arxiv.org/abs/2001.08327v4
• Date: Thu, 16 Sep 2021 16:11:40 GMT
• Title: The Reciprocal Bayesian LASSO
• Authors: Himel Mallick, Rahim Alhamzawi, Erina Paul, Vladimir Svetnik
• Abstract summary: We consider a fully Bayesian formulation of the rLASSO problem, which is based on the observation that the rLASSO estimate for linear regression parameters can be interpreted as a Bayesian posterior mode estimate. On simulated and real datasets, we show that the Bayesian formulation outperforms its classical cousin in estimation, prediction, and variable selection.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A reciprocal LASSO (rLASSO) regularization employs a decreasing penalty function as opposed to conventional penalization approaches that use increasing penalties on the coefficients, leading to stronger parsimony and superior model selection relative to traditional shrinkage methods. Here we consider a fully Bayesian formulation of the rLASSO problem, which is based on the observation that the rLASSO estimate for linear regression parameters can be interpreted as a Bayesian posterior mode estimate when the regression parameters are assigned independent inverse Laplace priors. Bayesian inference from this posterior is possible using an expanded hierarchy motivated by a scale mixture of double Pareto or truncated normal distributions. On simulated and real datasets, we show that the Bayesian formulation outperforms its classical cousin in estimation, prediction, and variable selection across a wide range of scenarios while offering the advantage of posterior inference. Finally, we discuss other variants of this new approach and provide a unified framework for variable selection using flexible reciprocal penalties. All methods described in this paper are publicly available as an R package at: https://github.com/himelmallick/BayesRecipe.

Related papers

• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate. We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Generalized Laplace Approximation [23.185126261153236]
  We introduce a unified theoretical framework to attribute Bayesian inconsistency to model misspecification and inadequate priors. We propose the generalized Laplace approximation, which involves a simple adjustment to the Hessian matrix of the regularized loss function. We assess the performance and properties of the generalized Laplace approximation on state-of-the-art neural networks and real-world datasets.
  arXiv  Detail & Related papers  (2024-05-22T11:11:42Z)
• Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance [52.093434664236014]
  Recent diffusion models provide a promising zero-shot solution to noisy linear inverse problems without retraining for specific inverse problems. Inspired by this finding, we propose to improve recent methods by using more principled covariance determined by maximum likelihood estimation.
  arXiv  Detail & Related papers  (2024-02-03T13:35:39Z)
• Variational Prediction [95.00085314353436]
  We present a technique for learning a variational approximation to the posterior predictive distribution using a variational bound. This approach can provide good predictive distributions without test time marginalization costs.
  arXiv  Detail & Related papers  (2023-07-14T18:19:31Z)
• Variational Laplace Autoencoders [53.08170674326728]
  Variational autoencoders employ an amortized inference model to approximate the posterior of latent variables. We present a novel approach that addresses the limited posterior expressiveness of fully-factorized Gaussian assumption. We also present a general framework named Variational Laplace Autoencoders (VLAEs) for training deep generative models.
  arXiv  Detail & Related papers  (2022-11-30T18:59:27Z)
• A flexible empirical Bayes approach to multiple linear regression and connections with penalized regression [8.663322701649454]
  We introduce a new empirical Bayes approach for large-scale multiple linear regression. Our approach combines two key ideas: the use of flexible "adaptive shrinkage" priors and variational approximations. We show that the posterior mean from our method solves a penalized regression problem.
  arXiv  Detail & Related papers  (2022-08-23T12:42:57Z)
• Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence [77.06203118175335]
  Variational Inference (VI) is a popular alternative to exact sampling in Bayesian inference. Importance sampling (IS) is often used to fine-tune and de-bias the estimates of approximate Bayesian inference procedures. We propose a novel combination of optimization and sampling techniques for approximate Bayesian inference.
  arXiv  Detail & Related papers  (2021-06-30T11:00:24Z)
• Ridge Regression Revisited: Debiasing, Thresholding and Bootstrap [4.142720557665472]
  ridge regression may be worth another look since -- after debiasing and thresholding -- it may offer some advantages over the Lasso. In this paper, we define a debiased and thresholded ridge regression method, and prove a consistency result and a Gaussian approximation theorem. In addition to estimation, we consider the problem of prediction, and present a novel, hybrid bootstrap algorithm tailored for prediction intervals.
  arXiv  Detail & Related papers  (2020-09-17T05:04:10Z)
• Slice Sampling for General Completely Random Measures [74.24975039689893]
  We present a novel Markov chain Monte Carlo algorithm for posterior inference that adaptively sets the truncation level using auxiliary slice variables. The efficacy of the proposed algorithm is evaluated on several popular nonparametric models.
  arXiv  Detail & Related papers  (2020-06-24T17:53:53Z)
• A Batch Normalized Inference Network Keeps the KL Vanishing Away [35.40781000297285]
  Variational Autoencoder (VAE) is widely used to approximate a model's posterior on latent variables. VAE often converges to a degenerated local optimum known as "posterior collapse"
  arXiv  Detail & Related papers  (2020-04-27T05:20:01Z)
• An Equivalence between Bayesian Priors and Penalties in Variational Inference [8.45602005745865]
  In machine learning, it is common to optimize the parameters of a probabilistic model, modulated by an ad hoc regularization term that penalizes some values of the parameters. We fully characterize the regularizers that can arise according to this procedure, and provide a systematic way to compute the prior corresponding to a given penalty.
  arXiv  Detail & Related papers  (2020-02-01T09:48:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.