Empirical Bayes Estimation for Lasso-Type Regularizers: Analysis of Automatic Relevance Determination

• URL: http://arxiv.org/abs/2501.11280v3
• Date: Tue, 29 Apr 2025 01:29:47 GMT
• Title: Empirical Bayes Estimation for Lasso-Type Regularizers: Analysis of Automatic Relevance Determination
• Authors: Tsukasa Yoshida, Kazuho Watanabe,
• Abstract summary: This paper focuses on linear regression models with non-conjugate sparsity-inducing regularizers such as lasso and group lasso.<n>We derive the empirical Bayes estimators for the group lasso regularized linear regression models with limited parameters.
• Score: 0.21485350418225244
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper focuses on linear regression models with non-conjugate sparsity-inducing regularizers such as lasso and group lasso. Although the empirical Bayes approach enables us to estimate the regularization parameter, little is known on the properties of the estimators. In particular, many aspects regarding the specific conditions under which the mechanism of automatic relevance determination (ARD) occurs remain unexplained. In this paper, we derive the empirical Bayes estimators for the group lasso regularized linear regression models with limited parameters. It is shown that the estimators diverge under a specific condition, giving rise to the ARD mechanism. We also prove that empirical Bayes methods can produce the ARD mechanism in general regularized linear regression models and clarify the conditions under which models such as ridge, lasso, and group lasso can do so.

Related papers

• Induced Covariance for Causal Discovery in Linear Sparse Structures [55.2480439325792]
  Causal models seek to unravel the cause-effect relationships among variables from observed data. This paper introduces a novel causal discovery algorithm designed for settings in which variables exhibit linearly sparse relationships.
  arXiv  Detail & Related papers  (2024-10-02T04:01:38Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regression [0.0]
  This study explores the predictive properties of over parameterized nonlinear regression within the Bayesian framework. Posterior contraction is established for generalized linear and single-neuron models with Lipschitz continuous activation functions. The proposed method was validated via numerical simulations and a real data application.
  arXiv  Detail & Related papers  (2024-04-06T04:22:48Z)
• Variational Inference of Parameters in Opinion Dynamics Models [9.51311391391997]
  This work uses variational inference to estimate the parameters of an opinion dynamics ABM. We transform the inference process into an optimization problem suitable for automatic differentiation. Our approach estimates both macroscopic (bounded confidence intervals and backfire thresholds) and microscopic ($200$ categorical, agent-level roles) more accurately than simulation-based and MCMC methods.
  arXiv  Detail & Related papers  (2024-03-08T14:45:18Z)
• Substitute adjustment via recovery of latent variables [2.973331166114387]
  The deconfounder was proposed as a method for estimating causal parameters in a context with multiple causes and unobserved confounding. We disentangle the causal interpretation from the statistical estimation problem and show that the deconfounder in general estimates adjusted regression target parameters. Our results support that when the latent variable model of the regressors hold, substitute adjustment is a viable method for adjusted regression.
  arXiv  Detail & Related papers  (2024-03-01T00:23:44Z)
• Conformal Approach To Gaussian Process Surrogate Evaluation With Coverage Guarantees [47.22930583160043]
  We propose a method for building adaptive cross-conformal prediction intervals. The resulting conformal prediction intervals exhibit a level of adaptivity akin to Bayesian credibility sets. The potential applicability of the method is demonstrated in the context of surrogate modeling of an expensive-to-evaluate simulator of the clogging phenomenon in steam generators of nuclear reactors.
  arXiv  Detail & Related papers  (2024-01-15T14:45:18Z)
• Bayesian Renormalization [68.8204255655161]
  We present a fully information theoretic approach to renormalization inspired by Bayesian statistical inference. The main insight of Bayesian Renormalization is that the Fisher metric defines a correlation length that plays the role of an emergent RG scale. We provide insight into how the Bayesian Renormalization scheme relates to existing methods for data compression and data generation.
  arXiv  Detail & Related papers  (2023-05-17T18:00:28Z)
• Generalized generalized linear models: Convex estimation and online bounds [11.295523372922533]
  We introduce inequalities in a class of models (GL-based) models (GGLM) The proposed approach uses the operator-based approach to overcome nontemporal variation among models. We demonstrate the performance using numerical simulations and a real data example for incidents.
  arXiv  Detail & Related papers  (2023-04-26T19:19:42Z)
• On LASSO for High Dimensional Predictive Regression [0.0]
  This paper examines LASSO, a widely-used $L_1$-penalized regression method, in high dimensional linear predictive regressions. The consistency of LASSO is contingent upon two key components: the deviation bound of the cross product of the regressors and the error term. Using machine learning and macroeconomic domain expertise, LASSO demonstrates strong performance in forecasting the unemployment rate.
  arXiv  Detail & Related papers  (2022-12-14T06:14:58Z)
• The Implicit Delta Method [61.36121543728134]
  In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of uncertainty. We show that the change in the evaluation due to regularization is consistent for the variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference.
  arXiv  Detail & Related papers  (2022-11-11T19:34:17Z)
• Time varying regression with hidden linear dynamics [74.9914602730208]
  We revisit a model for time-varying linear regression that assumes the unknown parameters evolve according to a linear dynamical system. Counterintuitively, we show that when the underlying dynamics are stable the parameters of this model can be estimated from data by combining just two ordinary least squares estimates.
  arXiv  Detail & Related papers  (2021-12-29T23:37:06Z)
• Estimation of Bivariate Structural Causal Models by Variational Gaussian Process Regression Under Likelihoods Parametrised by Normalising Flows [74.85071867225533]
  Causal mechanisms can be described by structural causal models. One major drawback of state-of-the-art artificial intelligence is its lack of explainability.
  arXiv  Detail & Related papers  (2021-09-06T14:52:58Z)
• Explainable AI by BAPC -- Before and After correction Parameter Comparison [0.0]
  A local surrogate for an AI-model correcting a simpler 'base' model is introduced representing an analytical method to yield explanations of AI-predictions. The AI-model approximates the residual error of the linear model and the explanations are formulated in terms of the change of the interpretable base model's parameters.
  arXiv  Detail & Related papers  (2021-03-12T09:03:51Z)
• The Variational Method of Moments [65.91730154730905]
  conditional moment problem is a powerful formulation for describing structural causal parameters in terms of observables. Motivated by a variational minimax reformulation of OWGMM, we define a very general class of estimators for the conditional moment problem. We provide algorithms for valid statistical inference based on the same kind of variational reformulations.
  arXiv  Detail & Related papers  (2020-12-17T07:21:06Z)
• Posterior Differential Regularization with f-divergence for Improving Model Robustness [95.05725916287376]
  We focus on methods that regularize the model posterior difference between clean and noisy inputs. We generalize the posterior differential regularization to the family of $f$-divergences. Our experiments show that regularizing the posterior differential with $f$-divergence can result in well-improved model robustness.
  arXiv  Detail & Related papers  (2020-10-23T19:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.