Empirical Bayes Estimation for Lasso-Type Regularizers: Analysis of Automatic Relevance Determination

• URL: http://arxiv.org/abs/2501.11280v2
• Date: Tue, 04 Feb 2025 19:01:43 GMT
• Title: Empirical Bayes Estimation for Lasso-Type Regularizers: Analysis of Automatic Relevance Determination
• Authors: Tsukasa Yoshida, Kazuho Watanabe,
• Abstract summary: This paper focuses on linear regression models with non-conjugate sparsity-inducing regularizers such as lasso and group lasso. We derive the empirical Bayes estimators for the group lasso regularized linear regression models with a limited number of parameters.
• Score: 0.21485350418225244
• License:
• Abstract: This paper focuses on linear regression models with non-conjugate sparsity-inducing regularizers such as lasso and group lasso. Although empirical Bayes approach enables us to estimate the regularization parameter, little is known on the properties of the estimators. In particular, there are many unexplained aspects regarding the specific conditions under which the mechanism of automatic relevance determination (ARD) occurs. In this paper, we derive the empirical Bayes estimators for the group lasso regularized linear regression models with a limited number of parameters. It is shown that the estimators diverge under a certain condition, giving rise to the ARD mechanism. We also prove that empirical Bayes methods can produce ARD mechanism in general regularized linear regression models and clarify the conditions under which models such as ridge, lasso, and group lasso can produce ARD mechanism.

Related papers

• Induced Covariance for Causal Discovery in Linear Sparse Structures [55.2480439325792]
  Causal models seek to unravel the cause-effect relationships among variables from observed data. This paper introduces a novel causal discovery algorithm designed for settings in which variables exhibit linearly sparse relationships.
  arXiv  Detail & Related papers  (2024-10-02T04:01:38Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Variational Inference of Parameters in Opinion Dynamics Models [9.51311391391997]
  This work uses variational inference to estimate the parameters of an opinion dynamics ABM. We transform the inference process into an optimization problem suitable for automatic differentiation. Our approach estimates both macroscopic (bounded confidence intervals and backfire thresholds) and microscopic ($200$ categorical, agent-level roles) more accurately than simulation-based and MCMC methods.
  arXiv  Detail & Related papers  (2024-03-08T14:45:18Z)
• Conformal Approach To Gaussian Process Surrogate Evaluation With Coverage Guarantees [47.22930583160043]
  We propose a method for building adaptive cross-conformal prediction intervals. The resulting conformal prediction intervals exhibit a level of adaptivity akin to Bayesian credibility sets. The potential applicability of the method is demonstrated in the context of surrogate modeling of an expensive-to-evaluate simulator of the clogging phenomenon in steam generators of nuclear reactors.
  arXiv  Detail & Related papers  (2024-01-15T14:45:18Z)
• Convex Parameter Estimation of Perturbed Multivariate Generalized Gaussian Distributions [18.95928707619676]
  We propose a convex formulation with well-established properties for MGGD parameters. The proposed framework is flexible as it combines a variety of regularizations for the precision matrix, the mean and perturbations. Experiments show a more accurate precision and covariance matrix estimation with similar performance for the mean vector parameter.
  arXiv  Detail & Related papers  (2023-12-12T18:08:04Z)
• Generalized generalized linear models: Convex estimation and online bounds [11.295523372922533]
  We introduce inequalities in a class of models (GL-based) models (GGLM) The proposed approach uses the operator-based approach to overcome nontemporal variation among models. We demonstrate the performance using numerical simulations and a real data example for incidents.
  arXiv  Detail & Related papers  (2023-04-26T19:19:42Z)
• On LASSO for High Dimensional Predictive Regression [0.0]
  This paper examines LASSO, a widely-used $L_1$-penalized regression method, in high dimensional linear predictive regressions. The consistency of LASSO is contingent upon two key components: the deviation bound of the cross product of the regressors and the error term. Using machine learning and macroeconomic domain expertise, LASSO demonstrates strong performance in forecasting the unemployment rate.
  arXiv  Detail & Related papers  (2022-12-14T06:14:58Z)
• Time varying regression with hidden linear dynamics [74.9914602730208]
  We revisit a model for time-varying linear regression that assumes the unknown parameters evolve according to a linear dynamical system. Counterintuitively, we show that when the underlying dynamics are stable the parameters of this model can be estimated from data by combining just two ordinary least squares estimates.
  arXiv  Detail & Related papers  (2021-12-29T23:37:06Z)
• Estimation of Bivariate Structural Causal Models by Variational Gaussian Process Regression Under Likelihoods Parametrised by Normalising Flows [74.85071867225533]
  Causal mechanisms can be described by structural causal models. One major drawback of state-of-the-art artificial intelligence is its lack of explainability.
  arXiv  Detail & Related papers  (2021-09-06T14:52:58Z)
• The Variational Method of Moments [65.91730154730905]
  conditional moment problem is a powerful formulation for describing structural causal parameters in terms of observables. Motivated by a variational minimax reformulation of OWGMM, we define a very general class of estimators for the conditional moment problem. We provide algorithms for valid statistical inference based on the same kind of variational reformulations.
  arXiv  Detail & Related papers  (2020-12-17T07:21:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.