DOODLER: Determining Out-Of-Distribution Likelihood from Encoder Reconstructions

• URL: http://arxiv.org/abs/2109.13237v1
• Date: Mon, 27 Sep 2021 14:54:55 GMT
• Title: DOODLER: Determining Out-Of-Distribution Likelihood from Encoder Reconstructions
• Authors: Jonathan S. Kent, Bo Li
• Abstract summary: This paper introduces and examines a novel methodology, DOODLER, for Out-Of-Distribution Detection. By training a Variational Auto-Encoder on the same data as another Deep Learning model, the VAE learns to accurately reconstruct In-Distribution (ID) inputs, but not to reconstruct OOD inputs. Unlike other work in the area, DOODLER requires only very weak assumptions about the existence of an OOD dataset, allowing for more realistic application.
• Score: 6.577622354490276
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deep Learning models possess two key traits that, in combination, make their use in the real world a risky prospect. One, they do not typically generalize well outside of the distribution for which they were trained, and two, they tend to exhibit confident behavior regardless of whether or not they are producing meaningful outputs. While Deep Learning possesses immense power to solve realistic, high-dimensional problems, these traits in concert make it difficult to have confidence in their real-world applications. To overcome this difficulty, the task of Out-Of-Distribution (OOD) Detection has been defined, to determine when a model has received an input from outside of the distribution for which it is trained to operate. This paper introduces and examines a novel methodology, DOODLER, for OOD Detection, which directly leverages the traits which result in its necessity. By training a Variational Auto-Encoder (VAE) on the same data as another Deep Learning model, the VAE learns to accurately reconstruct In-Distribution (ID) inputs, but not to reconstruct OOD inputs, meaning that its failure state can be used to perform OOD Detection. Unlike other work in the area, DOODLER requires only very weak assumptions about the existence of an OOD dataset, allowing for more realistic application. DOODLER also enables pixel-wise segmentations of input images by OOD likelihood, and experimental results show that it matches or outperforms methodologies that operate under the same constraints.

Related papers

• What If the Input is Expanded in OOD Detection? [77.37433624869857]
  Out-of-distribution (OOD) detection aims to identify OOD inputs from unknown classes. Various scoring functions are proposed to distinguish it from in-distribution (ID) data. We introduce a novel perspective, i.e., employing different common corruptions on the input space.
  arXiv  Detail & Related papers  (2024-10-24T06:47:28Z)
• The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection [75.65876949930258]
  Out-of-distribution (OOD) detection is essential for model trustworthiness. We show that the superior OOD detection performance of state-of-the-art methods is achieved by secretly sacrificing the OOD generalization ability.
  arXiv  Detail & Related papers  (2024-10-12T07:02:04Z)
• Can OOD Object Detectors Learn from Foundation Models? [56.03404530594071]
  Out-of-distribution (OOD) object detection is a challenging task due to the absence of open-set OOD data. Inspired by recent advancements in text-to-image generative models, we study the potential of generative models trained on large-scale open-set data to synthesize OOD samples. We introduce SyncOOD, a simple data curation method that capitalizes on the capabilities of large foundation models.
  arXiv  Detail & Related papers  (2024-09-08T17:28:22Z)
• Evaluating Reliability in Medical DNNs: A Critical Analysis of Feature and Confidence-Based OOD Detection [2.9049649065453336]
  OOD detection methods can be categorised as confidence-based (using the model's output layer for OOD detection) or feature-based (not using the output layer) We show that OOD artefacts can boost a model's softmax confidence in its predictions, due to correlations in training data among other factors. We also show that feature-based methods typically perform worse at distinguishing between inputs that lead to correct and incorrect predictions.
  arXiv  Detail & Related papers  (2024-08-30T15:02:22Z)
• Out-of-Distribution Learning with Human Feedback [26.398598663165636]
  This paper presents a novel framework for OOD learning with human feedback. Our framework capitalizes on the freely available unlabeled data in the wild. By exploiting human feedback, we enhance the robustness and reliability of machine learning models.
  arXiv  Detail & Related papers  (2024-08-14T18:49:27Z)
• Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources [73.28967478098107]
  Out-of-distribution (OOD) detection discerns OOD data where the predictor cannot make valid predictions as in-distribution (ID) data. It is typically hard to collect real out-of-distribution (OOD) data for training a predictor capable of discerning OOD patterns. We propose a data generation-based learning method named Auxiliary Task-based OOD Learning (ATOL) that can relieve the mistaken OOD generation.
  arXiv  Detail & Related papers  (2023-11-06T16:26:52Z)
• Unsupervised Out-of-Distribution Detection by Restoring Lossy Inputs with Variational Autoencoder [3.498694457257263]
  We propose a novel VAE-based score called Error Reduction (ER) for OOD detection. ER is based on a VAE that takes a lossy version of the training set as inputs and the original set as targets.
  arXiv  Detail & Related papers  (2023-09-05T09:42:15Z)
• Out-of-distribution Detection with Implicit Outlier Transformation [72.73711947366377]
  Outlier exposure (OE) is powerful in out-of-distribution (OOD) detection. We propose a novel OE-based approach that makes the model perform well for unseen OOD situations.
  arXiv  Detail & Related papers  (2023-03-09T04:36:38Z)
• Using Semantic Information for Defining and Detecting OOD Inputs [3.9577682622066264]
  Out-of-distribution (OOD) detection has received some attention recently. We demonstrate that the current detectors inherit the biases in the training dataset. This can render the current OOD detectors impermeable to inputs lying outside the training distribution but with the same semantic information. We perform OOD detection on semantic information extracted from the training data of MNIST and COCO datasets.
  arXiv  Detail & Related papers  (2023-02-21T21:31:20Z)
• Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities [104.02531442035483]
  The goal of this paper is to recognize common objectives as well as to identify the implicit scoring functions of different OOD detection methods. We show that binary discrimination between in- and (different) out-distributions is equivalent to several distinct formulations of the OOD detection problem. We also show that the confidence loss which is used by Outlier Exposure has an implicit scoring function which differs in a non-trivial fashion from the theoretically optimal scoring function.
  arXiv  Detail & Related papers  (2022-06-20T16:32:49Z)
• OODformer: Out-Of-Distribution Detection Transformer [15.17006322500865]
  In real-world safety-critical applications, it is important to be aware if a new data point is OOD. This paper proposes a first-of-its-kind OOD detection architecture named OODformer.
  arXiv  Detail & Related papers  (2021-07-19T15:46:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.