Entropic Issues in Likelihood-Based OOD Detection

• URL: http://arxiv.org/abs/2109.10794v2
• Date: Wed, 14 Jun 2023 18:02:07 GMT
• Title: Entropic Issues in Likelihood-Based OOD Detection
• Authors: Anthony L. Caterini, Gabriel Loaiza-Ganem
• Abstract summary: We provide a novel perspective on the phenomenon, decomposing the average likelihood into a KL divergence term and an entropy term. We argue that the latter can explain the curious OOD behaviour, suppressing likelihood values on datasets with higher entropy.
• Score: 14.612834877367046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep generative models trained by maximum likelihood remain very popular methods for reasoning about data probabilistically. However, it has been observed that they can assign higher likelihoods to out-of-distribution (OOD) data than in-distribution data, thus calling into question the meaning of these likelihood values. In this work we provide a novel perspective on this phenomenon, decomposing the average likelihood into a KL divergence term and an entropy term. We argue that the latter can explain the curious OOD behaviour mentioned above, suppressing likelihood values on datasets with higher entropy. Although our idea is simple, we have not seen it explored yet in the literature. This analysis provides further explanation for the success of OOD detection methods based on likelihood ratios, as the problematic entropy term cancels out in expectation. Finally, we discuss how this observation relates to recent success in OOD detection with manifold-supported models, for which the above decomposition does not hold directly.

Related papers

• Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection [63.93728560200819]
  Unsupervised out-of-distribution (U-OOD) detection is to identify data samples with a detector trained solely on unlabeled in-distribution (ID) data. Recent studies have developed various detectors based on DGMs to move beyond likelihood. We apply two techniques for each direction, specifically post-hoc prior and dataset entropy-mutual calibration. Experimental results demonstrate that the Resultant could be a new state-of-the-art U-OOD detector.
  arXiv  Detail & Related papers  (2024-09-05T02:58:13Z)
• Estimating Causal Effects from Learned Causal Networks [56.14597641617531]
  We propose an alternative paradigm for answering causal-effect queries over discrete observable variables. We learn the causal Bayesian network and its confounding latent variables directly from the observational data. We show that this emphmodel completion learning approach can be more effective than estimand approaches.
  arXiv  Detail & Related papers  (2024-08-26T08:39:09Z)
• A Geometric Explanation of the Likelihood OOD Detection Paradox [19.205693812937422]
  We show that high-likelihood regions will not be generated if they contain minimal probability mass. We propose a method for OOD detection which pairs the likelihoods and LID estimates obtained from a pre-trained DGM.
  arXiv  Detail & Related papers  (2024-03-27T18:02:49Z)
• Principled Knowledge Extrapolation with GANs [92.62635018136476]
  We study counterfactual synthesis from a new perspective of knowledge extrapolation. We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem. Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
  arXiv  Detail & Related papers  (2022-05-21T08:39:42Z)
• Understanding Failures in Out-of-Distribution Detection with Deep Generative Models [22.11487118547924]
  We prove that no method can guarantee performance beyond random chance without assumptions on which out-distributions are relevant. We highlight the consequences implied by assuming support overlap between in- and out-distributions. Our results suggest that estimation error is a more plausible explanation than the misalignment between likelihood-based OOD detection and out-distributions of interest.
  arXiv  Detail & Related papers  (2021-07-14T18:00:11Z)
• A statistical theory of out-of-distribution detection [26.928175726673615]
  We introduce a principled approach to detecting out-of-distribution data by exploiting a connection to data curation. In data curation, we exclude ambiguous or difficult-to-classify input points from the dataset, and these excluded points are by definition OOD. We can therefore obtain the likelihood for OOD points by using a principled generative model of data-curation.
  arXiv  Detail & Related papers  (2021-02-24T12:35:43Z)
• Perfect density models cannot guarantee anomaly detection [9.64157020816848]
  In this paper, we take a closer look at the behavior of distribution densities and show that these quantities carry less meaningful information than previously thought. We conclude that the use of these likelihoods for out-of-distribution detection relies on strong and implicit hypotheses.
  arXiv  Detail & Related papers  (2020-12-07T15:50:11Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)
• Density of States Estimation for Out-of-Distribution Detection [69.90130863160384]
  DoSE is the density of states estimator. We demonstrate DoSE's state-of-the-art performance against other unsupervised OOD detectors.
  arXiv  Detail & Related papers  (2020-06-16T16:06:25Z)
• Maximizing Information Gain in Partially Observable Environments via Prediction Reward [64.24528565312463]
  This paper tackles the challenge of using belief-based rewards for a deep RL agent. We derive the exact error between negative entropy and the expected prediction reward. This insight provides theoretical motivation for several fields using prediction rewards.
  arXiv  Detail & Related papers  (2020-05-11T08:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.