Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis

• URL: http://arxiv.org/abs/2107.02568v1
• Date: Tue, 6 Jul 2021 12:10:09 GMT
• Title: Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis
• Authors: Christoph Berger, Magdalini Paschali, Ben Glocker, Konstantinos Kamnitsas
• Abstract summary: We assess the capability of various state-of-the-art approaches for confidence-based OOD detection. First, we leverage a computer vision benchmark to reproduce and compare multiple OOD detection methods. We then evaluate their capabilities on the challenging task of disease classification using chest X-rays.
• Score: 17.398553230843717
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image classification models deployed in the real world may receive inputs outside the intended data distribution. For critical applications such as clinical decision making, it is important that a model can detect such out-of-distribution (OOD) inputs and express its uncertainty. In this work, we assess the capability of various state-of-the-art approaches for confidence-based OOD detection through a comparative study and in-depth analysis. First, we leverage a computer vision benchmark to reproduce and compare multiple OOD detection methods. We then evaluate their capabilities on the challenging task of disease classification using chest X-rays. Our study shows that high performance in a computer vision task does not directly translate to accuracy in a medical imaging task. We analyse factors that affect performance of the methods between the two tasks. Our results provide useful insights for developing the next generation of OOD detection methods.

Related papers

• Out-of-distribution Detection in Medical Image Analysis: A survey [12.778646136644399]
  Computer-aided diagnostics has benefited from the development of deep learning-based computer vision techniques. Traditional supervised deep learning methods assume that the test sample is drawn from the identical distribution as the training data. It is possible to encounter out-of-distribution samples in real-world clinical scenarios, which may cause silent failure in deep learning-based medical image analysis tasks.
  arXiv  Detail & Related papers  (2024-04-28T18:51:32Z)
• EndoOOD: Uncertainty-aware Out-of-distribution Detection in Capsule Endoscopy Diagnosis [11.82953216903558]
  Wireless capsule endoscopy (WCE) is a non-invasive diagnostic procedure that enables visualization of the gastrointestinal (GI) tract. Deep learning-based methods have shown effectiveness in disease screening using WCE data. Existing capsule endoscopy classification methods mostly rely on pre-defined categories.
  arXiv  Detail & Related papers  (2024-02-18T06:54:51Z)
• From Global to Local: Multi-scale Out-of-distribution Detection [129.37607313927458]
  Out-of-distribution (OOD) detection aims to detect "unknown" data whose labels have not been seen during the in-distribution (ID) training process. Recent progress in representation learning gives rise to distance-based OOD detection. We propose Multi-scale OOD DEtection (MODE), a first framework leveraging both global visual information and local region details.
  arXiv  Detail & Related papers  (2023-08-20T11:56:25Z)
• Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability [70.72426887518517]
  Out-of-distribution (OOD) detection is an indispensable aspect of secure AI when deploying machine learning models in real-world applications. We propose a novel method, Unleashing Mask, which aims to restore the OOD discriminative capabilities of the well-trained model with ID data. Our method utilizes a mask to figure out the memorized atypical samples, and then finetune the model or prune it with the introduced mask to forget them.
  arXiv  Detail & Related papers  (2023-06-06T14:23:34Z)
• LINe: Out-of-Distribution Detection by Leveraging Important Neurons [15.797257361788812]
  We introduce a new aspect for analyzing the difference in model outputs between in-distribution data and OOD data. We propose a novel method, Leveraging Important Neurons (LINe), for post-hoc Out of distribution detection.
  arXiv  Detail & Related papers  (2023-03-24T13:49:05Z)
• Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective [55.45202687256175]
  Out-of-distribution (OOD) detection methods assume that they have test ground truths, i.e., whether individual test samples are in-distribution (IND) or OOD. In this paper, we are the first to introduce the unsupervised evaluation problem in OOD detection. We propose three methods to compute Gscore as an unsupervised indicator of OOD detection performance.
  arXiv  Detail & Related papers  (2023-02-16T13:34:35Z)
• Energy-based Out-of-Distribution Detection for Graph Neural Networks [76.0242218180483]
  We propose a simple, powerful and efficient OOD detection model for GNN-based learning on graphs, which we call GNNSafe. GNNSafe achieves up to $17.0%$ AUROC improvement over state-of-the-arts and it could serve as simple yet strong baselines in such an under-developed area.
  arXiv  Detail & Related papers  (2023-02-06T16:38:43Z)
• Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need [52.88953913542445]
  We find surprisingly that simply using reconstruction-based methods could boost the performance of OOD detection significantly. We take Masked Image Modeling as a pretext task for our OOD detection framework (MOOD)
  arXiv  Detail & Related papers  (2023-02-06T08:24:41Z)
• Performance or Trust? Why Not Both. Deep AUC Maximization with Self-Supervised Learning for COVID-19 Chest X-ray Classifications [72.52228843498193]
  In training deep learning models, a compromise often must be made between performance and trust. In this work, we integrate a new surrogate loss with self-supervised learning for computer-aided screening of COVID-19 patients.
  arXiv  Detail & Related papers  (2021-12-14T21:16:52Z)
• Explaining medical AI performance disparities across sites with confounder Shapley value analysis [8.785345834486057]
  Multi-site evaluations are key to diagnosing such disparities. Our framework provides a method for quantifying the marginal and cumulative effect of each type of bias on the overall performance difference. We demonstrate its usefulness in a case study of a deep learning model trained to detect the presence of pneumothorax.
  arXiv  Detail & Related papers  (2021-11-12T18:54:10Z)
• Practical Evaluation of Out-of-Distribution Detection Methods for Image Classification [22.26009759606856]
  In this paper, we experimentally evaluate the performance of representative OOD detection methods for three scenarios. The results show that differences in scenarios and datasets alter the relative performance among the methods. Our results can also be used as a guide for the selection of OOD detection methods.
  arXiv  Detail & Related papers  (2021-01-07T09:28:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.