Robust Generalised Bayesian Inference for Intractable Likelihoods

• URL: http://arxiv.org/abs/2104.07359v1
• Date: Thu, 15 Apr 2021 10:31:22 GMT
• Title: Robust Generalised Bayesian Inference for Intractable Likelihoods
• Authors: Takuo Matsubara, Jeremias Knoblauch, Fran\c{c}ois-Xavier Briol, Chris. J. Oates
• Abstract summary: We consider generalised Bayesian inference with a Stein discrepancy as a loss function. This is motivated by applications in which the likelihood contains an intractable normalisation constant. We show consistency, normality and bias-robustness of the posterior, highlighting how these properties are impacted by the choice of Stein discrepancy.
• Score: 9.77823546576708
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generalised Bayesian inference updates prior beliefs using a loss function, rather than a likelihood, and can therefore be used to confer robustness against possible misspecification of the likelihood. Here we consider generalised Bayesian inference with a Stein discrepancy as a loss function, motivated by applications in which the likelihood contains an intractable normalisation constant. In this context, the Stein discrepancy circumvents evaluation of the normalisation constant and produces generalised posteriors that are either closed form or accessible using standard Markov chain Monte Carlo. On a theoretical level, we show consistency, asymptotic normality, and bias-robustness of the generalised posterior, highlighting how these properties are impacted by the choice of Stein discrepancy. Then, we provide numerical experiments on a range of intractable distributions, including applications to kernel-based exponential family models and non-Gaussian graphical models.

Related papers

• SoftCVI: Contrastive variational inference with self-generated soft labels [2.5398014196797614]
  Variational inference and Markov chain Monte Carlo methods are the predominant tools for this task. We introduce Soft Contrastive Variational Inference (SoftCVI), which allows a family of variational objectives to be derived through a contrastive estimation framework. We find that SoftCVI can be used to form objectives which are stable to train and mass-covering, frequently outperforming inference with other variational approaches.
  arXiv  Detail & Related papers  (2024-07-22T14:54:12Z)
• Generalized Laplace Approximation [23.185126261153236]
  We introduce a unified theoretical framework to attribute Bayesian inconsistency to model misspecification and inadequate priors. We propose the generalized Laplace approximation, which involves a simple adjustment to the Hessian matrix of the regularized loss function. We assess the performance and properties of the generalized Laplace approximation on state-of-the-art neural networks and real-world datasets.
  arXiv  Detail & Related papers  (2024-05-22T11:11:42Z)
• Bayesian Renormalization [68.8204255655161]
  We present a fully information theoretic approach to renormalization inspired by Bayesian statistical inference. The main insight of Bayesian Renormalization is that the Fisher metric defines a correlation length that plays the role of an emergent RG scale. We provide insight into how the Bayesian Renormalization scheme relates to existing methods for data compression and data generation.
  arXiv  Detail & Related papers  (2023-05-17T18:00:28Z)
• Large deviations rates for stochastic gradient descent with strongly convex functions [11.247580943940916]
  We provide a formal framework for the study of general high probability bounds with gradient descent. We find an upper large deviations bound for SGD with strongly convex functions.
  arXiv  Detail & Related papers  (2022-11-02T09:15:26Z)
• On the Importance of Gradient Norm in PAC-Bayesian Bounds [92.82627080794491]
  We propose a new generalization bound that exploits the contractivity of the log-Sobolev inequalities. We empirically analyze the effect of this new loss-gradient norm term on different neural architectures.
  arXiv  Detail & Related papers  (2022-10-12T12:49:20Z)
• Generalised Bayesian Inference for Discrete Intractable Likelihood [9.331721990371769]
  This paper develops a novel generalised Bayesian inference procedure suitable for discrete intractable likelihood. Inspired by recent methodological advances for continuous data, the main idea is to update beliefs about model parameters using a discrete Fisher divergence. The result is a generalised posterior that can be sampled from using standard computational tools, such as Markov Monte Carlo.
  arXiv  Detail & Related papers  (2022-06-16T19:36:17Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• Deconfounded Score Method: Scoring DAGs with Dense Unobserved Confounding [101.35070661471124]
  We show that unobserved confounding leaves a characteristic footprint in the observed data distribution that allows for disentangling spurious and causal effects. We propose an adjusted score-based causal discovery algorithm that may be implemented with general-purpose solvers and scales to high-dimensional problems.
  arXiv  Detail & Related papers  (2021-03-28T11:07:59Z)
• Generic Error Bounds for the Generalized Lasso with Sub-Exponential Data [4.56877715768796]
  This work performs a non-asymptotic analysis of the generalized Lasso under the assumption of sub-exponential data. We show that the estimation error can be controlled by means of two complexity parameters that arise naturally from a generic-chaining-based proof strategy.
  arXiv  Detail & Related papers  (2020-04-11T10:39:48Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.