Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities

• URL: http://arxiv.org/abs/2206.09880v1
• Date: Mon, 20 Jun 2022 16:32:49 GMT
• Title: Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities
• Authors: Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, Matthias Hein
• Abstract summary: The goal of this paper is to recognize common objectives as well as to identify the implicit scoring functions of different OOD detection methods. We show that binary discrimination between in- and (different) out-distributions is equivalent to several distinct formulations of the OOD detection problem. We also show that the confidence loss which is used by Outlier Exposure has an implicit scoring function which differs in a non-trivial fashion from the theoretically optimal scoring function.
• Score: 104.02531442035483
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: It is an important problem in trustworthy machine learning to recognize out-of-distribution (OOD) inputs which are inputs unrelated to the in-distribution task. Many out-of-distribution detection methods have been suggested in recent years. The goal of this paper is to recognize common objectives as well as to identify the implicit scoring functions of different OOD detection methods. We focus on the sub-class of methods that use surrogate OOD data during training in order to learn an OOD detection score that generalizes to new unseen out-distributions at test time. We show that binary discrimination between in- and (different) out-distributions is equivalent to several distinct formulations of the OOD detection problem. When trained in a shared fashion with a standard classifier, this binary discriminator reaches an OOD detection performance similar to that of Outlier Exposure. Moreover, we show that the confidence loss which is used by Outlier Exposure has an implicit scoring function which differs in a non-trivial fashion from the theoretically optimal scoring function in the case where training and test out-distribution are the same, which again is similar to the one used when training an Energy-Based OOD detector or when adding a background class. In practice, when trained in exactly the same way, all these methods perform similarly.

Related papers

• What If the Input is Expanded in OOD Detection? [77.37433624869857]
  Out-of-distribution (OOD) detection aims to identify OOD inputs from unknown classes. Various scoring functions are proposed to distinguish it from in-distribution (ID) data. We introduce a novel perspective, i.e., employing different common corruptions on the input space.
  arXiv  Detail & Related papers  (2024-10-24T06:47:28Z)
• Margin-bounded Confidence Scores for Out-of-Distribution Detection [2.373572816573706]
  We propose a novel method called Margin bounded Confidence Scores (MaCS) to address the nontrivial OOD detection problem. MaCS enlarges the disparity between ID and OOD scores, which in turn makes the decision boundary more compact. Experiments on various benchmark datasets for image classification tasks demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-09-22T05:40:25Z)
• Toward a Realistic Benchmark for Out-of-Distribution Detection [3.8038269045375515]
  We introduce a comprehensive benchmark for OOD detection based on ImageNet and Places365. Several techniques can be used to determine which classes should be considered in-distribution, yielding benchmarks with varying properties.
  arXiv  Detail & Related papers  (2024-04-16T11:29:43Z)
• EAT: Towards Long-Tailed Out-of-Distribution Detection [55.380390767978554]
  This paper addresses the challenging task of long-tailed OOD detection. The main difficulty lies in distinguishing OOD data from samples belonging to the tail classes. We propose two simple ideas: (1) Expanding the in-distribution class space by introducing multiple abstention classes, and (2) Augmenting the context-limited tail classes by overlaying images onto the context-rich OOD data.
  arXiv  Detail & Related papers  (2023-12-14T13:47:13Z)
• Beyond AUROC & co. for evaluating out-of-distribution detection performance [50.88341818412508]
  Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. We propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples.
  arXiv  Detail & Related papers  (2023-06-26T12:51:32Z)
• Understanding the properties and limitations of contrastive learning for Out-of-Distribution detection [3.2689702143620143]
  A popular approach to out-of-distribution (OOD) detection is based on a self-supervised learning technique referred to as contrastive learning. In this paper, we aim to understand the effectiveness and limitation of existing contrastive learning methods for OOD detection.
  arXiv  Detail & Related papers  (2022-11-06T17:33:29Z)
• Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition [80.07843757970923]
  We show that existing OOD detection methods suffer from significant performance degradation when the training set is long-tail distributed. We propose Partial and Asymmetric Supervised Contrastive Learning (PASCL), which explicitly encourages the model to distinguish between tail-class in-distribution samples and OOD samples. Our method outperforms previous state-of-the-art method by $1.29%$, $1.45%$, $0.69%$ anomaly detection false positive rate (FPR) and $3.24%$, $4.06%$, $7.89%$ in-distribution
  arXiv  Detail & Related papers  (2022-07-04T01:53:07Z)
• Multiple Testing Framework for Out-of-Distribution Detection [27.248375922343616]
  We study the problem of Out-of-Distribution (OOD) detection, that is, detecting whether a learning algorithm's output can be trusted at inference time. We propose a definition for the notion of OOD that includes both the input distribution and the learning algorithm, which provides insights for the construction of powerful tests for OOD detection.
  arXiv  Detail & Related papers  (2022-06-20T00:56:01Z)
• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law [78.10523907729642]
  VQA-CP has become the standard OOD benchmark for visual question answering. Most published methods rely on explicit knowledge of the construction of the OOD splits. We show that embarrassingly-simple methods, including one that generates answers at random, surpass the state of the art on some question types.
  arXiv  Detail & Related papers  (2020-05-19T06:45:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.