Confidence-based Out-of-Distribution Detection: A Comparative Study and
Analysis
- URL: http://arxiv.org/abs/2107.02568v1
- Date: Tue, 6 Jul 2021 12:10:09 GMT
- Title: Confidence-based Out-of-Distribution Detection: A Comparative Study and
Analysis
- Authors: Christoph Berger, Magdalini Paschali, Ben Glocker, Konstantinos
Kamnitsas
- Abstract summary: We assess the capability of various state-of-the-art approaches for confidence-based OOD detection.
First, we leverage a computer vision benchmark to reproduce and compare multiple OOD detection methods.
We then evaluate their capabilities on the challenging task of disease classification using chest X-rays.
- Score: 17.398553230843717
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Image classification models deployed in the real world may receive inputs
outside the intended data distribution. For critical applications such as
clinical decision making, it is important that a model can detect such
out-of-distribution (OOD) inputs and express its uncertainty. In this work, we
assess the capability of various state-of-the-art approaches for
confidence-based OOD detection through a comparative study and in-depth
analysis. First, we leverage a computer vision benchmark to reproduce and
compare multiple OOD detection methods. We then evaluate their capabilities on
the challenging task of disease classification using chest X-rays. Our study
shows that high performance in a computer vision task does not directly
translate to accuracy in a medical imaging task. We analyse factors that affect
performance of the methods between the two tasks. Our results provide useful
insights for developing the next generation of OOD detection methods.
Related papers
- Action-OOD: An End-to-End Skeleton-Based Model for Robust Out-of-Distribution Human Action Detection [17.85872085904999]
Action-OOD is a novel end-to-end skeleton-based model for action detection.
We introduce an attention-based feature fusion block, which enhances the model's capability to recognize unknown classes.
We demonstrate the superior performance of our proposed approach compared to state-of-the-art methods.
arXiv Detail & Related papers (2024-05-31T05:49:37Z) - Out-of-distribution Detection in Medical Image Analysis: A survey [12.778646136644399]
Computer-aided diagnostics has benefited from the development of deep learning-based computer vision techniques.
Traditional supervised deep learning methods assume that the test sample is drawn from the identical distribution as the training data.
It is possible to encounter out-of-distribution samples in real-world clinical scenarios, which may cause silent failure in deep learning-based medical image analysis tasks.
arXiv Detail & Related papers (2024-04-28T18:51:32Z) - From Global to Local: Multi-scale Out-of-distribution Detection [129.37607313927458]
Out-of-distribution (OOD) detection aims to detect "unknown" data whose labels have not been seen during the in-distribution (ID) training process.
Recent progress in representation learning gives rise to distance-based OOD detection.
We propose Multi-scale OOD DEtection (MODE), a first framework leveraging both global visual information and local region details.
arXiv Detail & Related papers (2023-08-20T11:56:25Z) - Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection
Capability [70.72426887518517]
Out-of-distribution (OOD) detection is an indispensable aspect of secure AI when deploying machine learning models in real-world applications.
We propose a novel method, Unleashing Mask, which aims to restore the OOD discriminative capabilities of the well-trained model with ID data.
Our method utilizes a mask to figure out the memorized atypical samples, and then finetune the model or prune it with the introduced mask to forget them.
arXiv Detail & Related papers (2023-06-06T14:23:34Z) - LINe: Out-of-Distribution Detection by Leveraging Important Neurons [15.797257361788812]
We introduce a new aspect for analyzing the difference in model outputs between in-distribution data and OOD data.
We propose a novel method, Leveraging Important Neurons (LINe), for post-hoc Out of distribution detection.
arXiv Detail & Related papers (2023-03-24T13:49:05Z) - Background Matters: Enhancing Out-of-distribution Detection with Domain
Features [90.32910087103744]
OOD samples can be drawn from arbitrary distributions and exhibit deviations from in-distribution (ID) data in various dimensions.
Existing methods focus on detecting OOD samples based on the semantic features, while neglecting the other dimensions such as the domain features.
This paper proposes a novel generic framework that can learn the domain features from the ID training samples by a dense prediction approach.
arXiv Detail & Related papers (2023-03-15T16:12:14Z) - Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric
Perspective [55.45202687256175]
Out-of-distribution (OOD) detection methods assume that they have test ground truths, i.e., whether individual test samples are in-distribution (IND) or OOD.
In this paper, we are the first to introduce the unsupervised evaluation problem in OOD detection.
We propose three methods to compute Gscore as an unsupervised indicator of OOD detection performance.
arXiv Detail & Related papers (2023-02-16T13:34:35Z) - Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is
All You Need [52.88953913542445]
We find surprisingly that simply using reconstruction-based methods could boost the performance of OOD detection significantly.
We take Masked Image Modeling as a pretext task for our OOD detection framework (MOOD)
arXiv Detail & Related papers (2023-02-06T08:24:41Z) - Performance or Trust? Why Not Both. Deep AUC Maximization with
Self-Supervised Learning for COVID-19 Chest X-ray Classifications [72.52228843498193]
In training deep learning models, a compromise often must be made between performance and trust.
In this work, we integrate a new surrogate loss with self-supervised learning for computer-aided screening of COVID-19 patients.
arXiv Detail & Related papers (2021-12-14T21:16:52Z) - Explaining medical AI performance disparities across sites with
confounder Shapley value analysis [8.785345834486057]
Multi-site evaluations are key to diagnosing such disparities.
Our framework provides a method for quantifying the marginal and cumulative effect of each type of bias on the overall performance difference.
We demonstrate its usefulness in a case study of a deep learning model trained to detect the presence of pneumothorax.
arXiv Detail & Related papers (2021-11-12T18:54:10Z) - Practical Evaluation of Out-of-Distribution Detection Methods for Image
Classification [22.26009759606856]
In this paper, we experimentally evaluate the performance of representative OOD detection methods for three scenarios.
The results show that differences in scenarios and datasets alter the relative performance among the methods.
Our results can also be used as a guide for the selection of OOD detection methods.
arXiv Detail & Related papers (2021-01-07T09:28:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.