Reproducible Parameter Inference Using Bagged Posteriors

• URL: http://arxiv.org/abs/2311.02019v1
• Date: Fri, 3 Nov 2023 16:28:16 GMT
• Title: Reproducible Parameter Inference Using Bagged Posteriors
• Authors: Jonathan H. Huggins, Jeffrey W. Miller
• Abstract summary: Under model misspecification, it is known that Bayesian posteriors often do not properly quantify uncertainty about true or pseudo-true parameters. We consider the probability that two confidence sets constructed from independent data sets have nonempty overlap. We show that credible sets from the standard posterior can strongly violate this bound, particularly in high-dimensional settings.
• Score: 9.975422461924705
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Under model misspecification, it is known that Bayesian posteriors often do not properly quantify uncertainty about true or pseudo-true parameters. Even more fundamentally, misspecification leads to a lack of reproducibility in the sense that the same model will yield contradictory posteriors on independent data sets from the true distribution. To define a criterion for reproducible uncertainty quantification under misspecification, we consider the probability that two confidence sets constructed from independent data sets have nonempty overlap, and we establish a lower bound on this overlap probability that holds for any valid confidence sets. We prove that credible sets from the standard posterior can strongly violate this bound, particularly in high-dimensional settings (i.e., with dimension increasing with sample size), indicating that it is not internally coherent under misspecification. To improve reproducibility in an easy-to-use and widely applicable way, we propose to apply bagging to the Bayesian posterior ("BayesBag"'); that is, to use the average of posterior distributions conditioned on bootstrapped datasets. We motivate BayesBag from first principles based on Jeffrey conditionalization and show that the bagged posterior typically satisfies the overlap lower bound. Further, we prove a Bernstein--Von Mises theorem for the bagged posterior, establishing its asymptotic normal distribution. We demonstrate the benefits of BayesBag via simulation experiments and an application to crime rate prediction.

Related papers

• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Predictive variational inference: Learn the predictively optimal posterior distribution [1.7648680700685022]
  Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density. This framework applies to both likelihood-exact and likelihood-free models.
  arXiv  Detail & Related papers  (2024-10-18T19:44:57Z)
• Bayesian meta learning for trustworthy uncertainty quantification [3.683202928838613]
  We propose, Trust-Bayes, a novel optimization framework for Bayesian meta learning. We characterize the lower bounds of the probabilities of the ground truth being captured by the specified intervals. We analyze the sample complexity with respect to the feasible probability for trustworthy uncertainty quantification.
  arXiv  Detail & Related papers  (2024-07-27T15:56:12Z)
• Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probability [50.44439018155837]
  We propose to include a calibration term directly into the training objective of the neural model. By introducing a relaxation of the classical formulation of calibration error we enable end-to-end backpropagation. It is directly applicable to existing computational pipelines allowing reliable black-box posterior inference.
  arXiv  Detail & Related papers  (2023-10-20T10:20:45Z)
• Sequential Gibbs Posteriors with Applications to Principal Component Analysis [8.90721241624138]
  Gibbs posteriors provide a principled framework for likelihood-free Bayesian inference. But in many situations, including a single tuning parameter inevitably leads to poor uncertainty quantification. We propose a sequential extension to Gibbs posteriors to address this problem.
  arXiv  Detail & Related papers  (2023-10-19T16:36:18Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)
• Sequential Predictive Two-Sample and Independence Testing [114.4130718687858]
  We study the problems of sequential nonparametric two-sample and independence testing. We build upon the principle of (nonparametric) testing by betting.
  arXiv  Detail & Related papers  (2023-04-29T01:30:33Z)
• Robust Generalised Bayesian Inference for Intractable Likelihoods [9.77823546576708]
  We consider generalised Bayesian inference with a Stein discrepancy as a loss function. This is motivated by applications in which the likelihood contains an intractable normalisation constant. We show consistency, normality and bias-robustness of the posterior, highlighting how these properties are impacted by the choice of Stein discrepancy.
  arXiv  Detail & Related papers  (2021-04-15T10:31:22Z)
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification [58.03725169462616]
  We show theoretically that over-parametrization is not the only reason for over-confidence. We prove that logistic regression is inherently over-confident, in the realizable, under-parametrized setting. Perhaps surprisingly, we also show that over-confidence is not always the case.
  arXiv  Detail & Related papers  (2021-02-15T21:38:09Z)
• Distribution-free binary classification: prediction sets, confidence intervals and calibration [106.50279469344937]
  We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting. We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
  arXiv  Detail & Related papers  (2020-06-18T14:17:29Z)
• Distributionally Robust Bayesian Quadrature Optimization [60.383252534861136]
  We study BQO under distributional uncertainty in which the underlying probability distribution is unknown except for a limited set of its i.i.d. samples. A standard BQO approach maximizes the Monte Carlo estimate of the true expected objective given the fixed sample set. We propose a novel posterior sampling based algorithm, namely distributionally robust BQO (DRBQO) for this purpose.
  arXiv  Detail & Related papers  (2020-01-19T12:00:33Z)
• A New Analysis of Differential Privacy's Generalization Guarantees [11.485744204944627]
  We show that any mechanism for answering adaptively chosen statistical queries that is differentially private and sample-accurate is also accurate out-of-sample. Our new proof is elementary and gives structural insights that we expect will be useful elsewhere.
  arXiv  Detail & Related papers  (2019-09-09T00:49:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.