Falsehoods that ML researchers believe about OOD detection

• URL: http://arxiv.org/abs/2210.12767v1
• Date: Sun, 23 Oct 2022 16:21:54 GMT
• Title: Falsehoods that ML researchers believe about OOD detection
• Authors: Andi Zhang, Damon Wischik
• Abstract summary: We list some falsehoods that machine learning researchers believe about density-based OOD detection. We propose a framework, the OOD proxy framework, to unify these methods.
• Score: 0.24801933141734633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modelling the density $p(x)$ by probabilistic generative models is an intuitive way to detect out-of-distribution (OOD) data, but it fails in the deep learning context. In this paper, we list some falsehoods that machine learning researchers believe about density-based OOD detection. Many recent works have proposed likelihood-ratio-based methods to `fix' this issue. We propose a framework, the OOD proxy framework, to unify these methods, and we argue that likelihood ratio is a principled method for OOD detection and not a mere `fix'. Finally, we discuss the relationship between domain detection and semantics.

Related papers

• The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection [75.65876949930258]
  Out-of-distribution (OOD) detection is essential for model trustworthiness. We show that the superior OOD detection performance of state-of-the-art methods is achieved by secretly sacrificing the OOD generalization ability.
  arXiv  Detail & Related papers  (2024-10-12T07:02:04Z)
• A Geometric Explanation of the Likelihood OOD Detection Paradox [19.205693812937422]
  We show that high-likelihood regions will not be generated if they contain minimal probability mass. We propose a method for OOD detection which pairs the likelihoods and LID estimates obtained from a pre-trained DGM.
  arXiv  Detail & Related papers  (2024-03-27T18:02:49Z)
• Model-free Test Time Adaptation for Out-Of-Distribution Detection [62.49795078366206]
  We propose a Non-Parametric Test Time textbfAdaptation framework for textbfDistribution textbfDetection (abbr) abbr utilizes online test samples for model adaptation during testing, enhancing adaptability to changing data distributions. We demonstrate the effectiveness of abbr through comprehensive experiments on multiple OOD detection benchmarks.
  arXiv  Detail & Related papers  (2023-11-28T02:00:47Z)
• FLatS: Principled Out-of-Distribution Detection with Feature-Based Likelihood Ratio Score [2.9914612342004503]
  FLatS is a principled solution for OOD detection based on likelihood ratio. We demonstrate that FLatS can serve as a general framework capable of enhancing other OOD detection methods. Experiments show that FLatS establishes a new SOTA on popular benchmarks.
  arXiv  Detail & Related papers  (2023-10-08T09:16:46Z)
• Beyond AUROC & co. for evaluating out-of-distribution detection performance [50.88341818412508]
  Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. We propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples.
  arXiv  Detail & Related papers  (2023-06-26T12:51:32Z)
• Distribution Calibration for Out-of-Domain Detection with Bayesian Approximation [35.34001858858684]
  Out-of-Domain (OOD) detection is a key component in a task-oriented dialog system. Previous softmax-based detection algorithms are proved to be overconfident for OOD samples. We propose a Bayesian OOD detection framework to calibrate distribution uncertainty using Monte-Carlo Dropout.
  arXiv  Detail & Related papers  (2022-09-14T13:04:09Z)
• No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets [69.725266027309]
  Out-of-distribution detection is an important component of reliable ML systems. In this work, we show that none of these methods are inherently better at OOD detection than others on a standardized set of 16 pairs. We also show that a method outperforming another on a certain (ID, OOD) pair may not do so in a low-data regime.
  arXiv  Detail & Related papers  (2021-09-12T16:35:00Z)
• Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection [76.39067237772286]
  We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
  arXiv  Detail & Related papers  (2020-12-10T16:55:13Z)
• Robust Out-of-distribution Detection for Neural Networks [51.19164318924997]
  We show that existing detection mechanisms can be extremely brittle when evaluating on in-distribution and OOD inputs. We propose an effective algorithm called ALOE, which performs robust training by exposing the model to both adversarially crafted inlier and outlier examples.
  arXiv  Detail & Related papers  (2020-03-21T17:46:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.