Reproducible Parameter Inference Using Bagged Posteriors
- URL: http://arxiv.org/abs/2311.02019v1
- Date: Fri, 3 Nov 2023 16:28:16 GMT
- Title: Reproducible Parameter Inference Using Bagged Posteriors
- Authors: Jonathan H. Huggins, Jeffrey W. Miller
- Abstract summary: Under model misspecification, it is known that Bayesian posteriors often do not properly quantify uncertainty about true or pseudo-true parameters.
We consider the probability that two confidence sets constructed from independent data sets have nonempty overlap.
We show that credible sets from the standard posterior can strongly violate this bound, particularly in high-dimensional settings.
- Score: 9.975422461924705
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Under model misspecification, it is known that Bayesian posteriors often do
not properly quantify uncertainty about true or pseudo-true parameters. Even
more fundamentally, misspecification leads to a lack of reproducibility in the
sense that the same model will yield contradictory posteriors on independent
data sets from the true distribution. To define a criterion for reproducible
uncertainty quantification under misspecification, we consider the probability
that two confidence sets constructed from independent data sets have nonempty
overlap, and we establish a lower bound on this overlap probability that holds
for any valid confidence sets. We prove that credible sets from the standard
posterior can strongly violate this bound, particularly in high-dimensional
settings (i.e., with dimension increasing with sample size), indicating that it
is not internally coherent under misspecification. To improve reproducibility
in an easy-to-use and widely applicable way, we propose to apply bagging to the
Bayesian posterior ("BayesBag"'); that is, to use the average of posterior
distributions conditioned on bootstrapped datasets. We motivate BayesBag from
first principles based on Jeffrey conditionalization and show that the bagged
posterior typically satisfies the overlap lower bound. Further, we prove a
Bernstein--Von Mises theorem for the bagged posterior, establishing its
asymptotic normal distribution. We demonstrate the benefits of BayesBag via
simulation experiments and an application to crime rate prediction.
Related papers
- Calibrating Neural Simulation-Based Inference with Differentiable
Coverage Probability [50.44439018155837]
We propose to include a calibration term directly into the training objective of the neural model.
By introducing a relaxation of the classical formulation of calibration error we enable end-to-end backpropagation.
It is directly applicable to existing computational pipelines allowing reliable black-box posterior inference.
arXiv Detail & Related papers (2023-10-20T10:20:45Z) - Sequential Gibbs Posteriors with Applications to Principal Component
Analysis [8.90721241624138]
Gibbs posteriors provide a principled framework for likelihood-free Bayesian inference.
But in many situations, including a single tuning parameter inevitably leads to poor uncertainty quantification.
We propose a sequential extension to Gibbs posteriors to address this problem.
arXiv Detail & Related papers (2023-10-19T16:36:18Z) - When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
Cascades are a classical strategy to enable inference cost to vary adaptively across samples.
A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction.
Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
arXiv Detail & Related papers (2023-07-06T04:13:57Z) - Sequential Predictive Two-Sample and Independence Testing [114.4130718687858]
We study the problems of sequential nonparametric two-sample and independence testing.
We build upon the principle of (nonparametric) testing by betting.
arXiv Detail & Related papers (2023-04-29T01:30:33Z) - Uncertainty Estimates of Predictions via a General Bias-Variance
Decomposition [7.811916700683125]
We introduce a bias-variance decomposition for proper scores, giving rise to the Bregman Information as the variance term.
We showcase the practical relevance of this decomposition on several downstream tasks, including model ensembles and confidence regions.
arXiv Detail & Related papers (2022-10-21T21:24:37Z) - On Uncertainty, Tempering, and Data Augmentation in Bayesian
Classification [47.13680267076843]
We show that explicitly accounting for aleatoric uncertainty significantly improves the performance of Bayesian neural networks.
We find that a cold posterior, tempered by a power greater than one, often more honestly reflects our beliefs about aleatoric uncertainty than no tempering.
arXiv Detail & Related papers (2022-03-30T17:17:50Z) - Robust Generalised Bayesian Inference for Intractable Likelihoods [9.77823546576708]
We consider generalised Bayesian inference with a Stein discrepancy as a loss function.
This is motivated by applications in which the likelihood contains an intractable normalisation constant.
We show consistency, normality and bias-robustness of the posterior, highlighting how these properties are impacted by the choice of Stein discrepancy.
arXiv Detail & Related papers (2021-04-15T10:31:22Z) - Don't Just Blame Over-parametrization for Over-confidence: Theoretical
Analysis of Calibration in Binary Classification [58.03725169462616]
We show theoretically that over-parametrization is not the only reason for over-confidence.
We prove that logistic regression is inherently over-confident, in the realizable, under-parametrized setting.
Perhaps surprisingly, we also show that over-confidence is not always the case.
arXiv Detail & Related papers (2021-02-15T21:38:09Z) - Distribution-free binary classification: prediction sets, confidence
intervals and calibration [106.50279469344937]
We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting.
We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning.
As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
arXiv Detail & Related papers (2020-06-18T14:17:29Z) - Distributionally Robust Bayesian Quadrature Optimization [60.383252534861136]
We study BQO under distributional uncertainty in which the underlying probability distribution is unknown except for a limited set of its i.i.d. samples.
A standard BQO approach maximizes the Monte Carlo estimate of the true expected objective given the fixed sample set.
We propose a novel posterior sampling based algorithm, namely distributionally robust BQO (DRBQO) for this purpose.
arXiv Detail & Related papers (2020-01-19T12:00:33Z) - A New Analysis of Differential Privacy's Generalization Guarantees [11.485744204944627]
We show that any mechanism for answering adaptively chosen statistical queries that is differentially private and sample-accurate is also accurate out-of-sample.
Our new proof is elementary and gives structural insights that we expect will be useful elsewhere.
arXiv Detail & Related papers (2019-09-09T00:49:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.