Quantification of Predictive Uncertainty via Inference-Time Sampling

• URL: http://arxiv.org/abs/2308.01731v1
• Date: Thu, 3 Aug 2023 12:43:21 GMT
• Title: Quantification of Predictive Uncertainty via Inference-Time Sampling
• Authors: Katar\'ina T\'othov\'a, \v{L}ubor Ladick\'y, Daniel Thul, Marc Pollefeys, Ender Konukoglu
• Abstract summary: We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
• Score: 57.749601811982096
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predictive variability due to data ambiguities has typically been addressed via construction of dedicated models with built-in probabilistic capabilities that are trained to predict uncertainty estimates as variables of interest. These approaches require distinct architectural components and training mechanisms, may include restrictive assumptions and exhibit overconfidence, i.e., high confidence in imprecise predictions. In this work, we propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions. It is architecture agnostic and can be applied to any feed-forward deterministic network without changes to the architecture or training procedure. Experiments on regression tasks on imaging and non-imaging input data show the method's ability to generate diverse and multi-modal predictive distributions, and a desirable correlation of the estimated uncertainty with the prediction error.

Related papers

• Model-agnostic variable importance for predictive uncertainty: an entropy-based approach [1.912429179274357]
  We show how existing methods in explainability can be extended to uncertainty-aware models. We demonstrate the utility of these approaches to understand both the sources of uncertainty and their impact on model performance.
  arXiv  Detail & Related papers  (2023-10-19T15:51:23Z)
• Invariant Probabilistic Prediction [45.90606906307022]
  We show that arbitrary distribution shifts do not, in general, admit invariant and robust probabilistic predictions. We propose a method to yield invariant probabilistic predictions, called IPP, and study the consistency of the underlying parameters.
  arXiv  Detail & Related papers  (2023-09-18T18:50:24Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Robust uncertainty estimates with out-of-distribution pseudo-inputs training [0.0]
  We propose to explicitly train the uncertainty predictor where we are not given data to make it reliable. As one cannot train without data, we provide mechanisms for generating pseudo-inputs in informative low-density regions of the input space. With a holistic evaluation, we demonstrate that this yields robust and interpretable predictions of uncertainty while retaining state-of-the-art performance on diverse tasks.
  arXiv  Detail & Related papers  (2022-01-15T17:15:07Z)
• Dense Uncertainty Estimation [62.23555922631451]
  In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation. We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
  arXiv  Detail & Related papers  (2021-10-13T01:23:48Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts [33.45069308137142]
  Posterior Network (PostNet) predicts an individual closed-form posterior distribution over predicted probabilites for any input sample. PostNet achieves state-of-the art results in OOD detection and in uncertainty calibration under dataset shifts.
  arXiv  Detail & Related papers  (2020-06-16T15:16:32Z)
• Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation [25.97333838935589]
  Uncertainty estimates are an important tool for helping autonomous agents or human decision makers understand and leverage predictive models. Existing algorithms can overestimate certainty, possibly yielding a false sense of confidence in the predictive model.
  arXiv  Detail & Related papers  (2020-02-29T20:31:04Z)
• Learning to Predict Error for MRI Reconstruction [67.76632988696943]
  We demonstrate that predictive uncertainty estimated by the current methods does not highly correlate with prediction error. We propose a novel method that estimates the target labels and magnitude of the prediction error in two steps.
  arXiv  Detail & Related papers  (2020-02-13T15:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.