Probabilistic Circuits That Know What They Don't Know

• URL: http://arxiv.org/abs/2302.06544v3
• Date: Mon, 12 Jun 2023 06:36:53 GMT
• Title: Probabilistic Circuits That Know What They Don't Know
• Abstract summary: Probabilistic circuits (PCs) are models that allow exact and tractable probabilistic inference. We show that PCs are in fact not robust to out-of-distribution (OOD) data, i.e., they don't know what they don't know. We propose tractable dropout inference (TDI), an inference procedure to estimate uncertainty by deriving an analytical solution to Monte Carlo dropout (MCD) through variance propagation.
• Score: 17.335642512433385
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Probabilistic circuits (PCs) are models that allow exact and tractable probabilistic inference. In contrast to neural networks, they are often assumed to be well-calibrated and robust to out-of-distribution (OOD) data. In this paper, we show that PCs are in fact not robust to OOD data, i.e., they don't know what they don't know. We then show how this challenge can be overcome by model uncertainty quantification. To this end, we propose tractable dropout inference (TDI), an inference procedure to estimate uncertainty by deriving an analytical solution to Monte Carlo dropout (MCD) through variance propagation. Unlike MCD in neural networks, which comes at the cost of multiple network evaluations, TDI provides tractable sampling-free uncertainty estimates in a single forward pass. TDI improves the robustness of PCs to distribution shift and OOD data, demonstrated through a series of experiments evaluating the classification confidence and uncertainty estimates on real-world data.

Related papers

• Cooperative Bayesian and variance networks disentangle aleatoric and epistemic uncertainties [0.0]
  Real-world data contains aleatoric uncertainty - irreducible noise arising from imperfect measurements or from incomplete knowledge about the data generation process.<n>Mean variance estimation (MVE) networks can learn this type of uncertainty but require ad-hoc regularization strategies to avoid overfitting.<n>We propose to train a variance network with a Bayesian neural network and demonstrate that the resulting model disentangles aleatoric and epistemic uncertainties while improving the mean estimation.
  arXiv  Detail & Related papers  (2025-05-05T15:50:52Z)
• Uncertainty-Aware Out-of-Distribution Detection with Gaussian Processes [13.246251147975192]
  Deep neural networks (DNNs) are often constructed under the closed-world assumption. OOD samples are not always available during the training phase in real-world applications. We propose a Gaussian-process-based OOD detection method to establish a decision boundary based on InD data only.
  arXiv  Detail & Related papers  (2024-12-30T12:57:31Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Robust uncertainty estimates with out-of-distribution pseudo-inputs training [0.0]
  We propose to explicitly train the uncertainty predictor where we are not given data to make it reliable. As one cannot train without data, we provide mechanisms for generating pseudo-inputs in informative low-density regions of the input space. With a holistic evaluation, we demonstrate that this yields robust and interpretable predictions of uncertainty while retaining state-of-the-art performance on diverse tasks.
  arXiv  Detail & Related papers  (2022-01-15T17:15:07Z)
• Statistical Testing for Efficient Out of Distribution Detection in Deep Neural Networks [26.0303701309125]
  This paper frames the Out Of Distribution (OOD) detection problem in Deep Neural Networks as a statistical hypothesis testing problem. We build on this framework to suggest a novel OOD procedure based on low-order statistics. Our method achieves comparable or better than state-of-the-art results on well-accepted OOD benchmarks without retraining the network parameters.
  arXiv  Detail & Related papers  (2021-02-25T16:14:47Z)
• A statistical theory of out-of-distribution detection [26.928175726673615]
  We introduce a principled approach to detecting out-of-distribution data by exploiting a connection to data curation. In data curation, we exclude ambiguous or difficult-to-classify input points from the dataset, and these excluded points are by definition OOD. We can therefore obtain the likelihood for OOD points by using a principled generative model of data-curation.
  arXiv  Detail & Related papers  (2021-02-24T12:35:43Z)
• Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection [76.39067237772286]
  We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
  arXiv  Detail & Related papers  (2020-12-10T16:55:13Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts [33.45069308137142]
  Posterior Network (PostNet) predicts an individual closed-form posterior distribution over predicted probabilites for any input sample. PostNet achieves state-of-the art results in OOD detection and in uncertainty calibration under dataset shifts.
  arXiv  Detail & Related papers  (2020-06-16T15:16:32Z)
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network [66.26231423824089]
  We propose a method for training a deterministic deep model that can find and reject out of distribution data points at test time with a single forward pass. We scale training in these with a novel loss function and centroid updating scheme and match the accuracy of softmax models.
  arXiv  Detail & Related papers  (2020-03-04T12:27:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.