A knee cannot have lung disease: out-of-distribution detection with in-distribution voting using the medical example of chest X-ray classification

• URL: http://arxiv.org/abs/2208.01077v2
• Date: Mon, 8 May 2023 08:44:15 GMT
• Title: A knee cannot have lung disease: out-of-distribution detection with in-distribution voting using the medical example of chest X-ray classification
• Authors: Alessandro Wollek, Theresa Willem, Michael Ingrisch, Bastian Sabel and Tobias Lasser
• Abstract summary: The study employed the commonly used chest X-ray classification model, CheXnet, trained on the chest X-ray 14 data set. To detect OOD data for multi-label classification, we proposed in-distribution voting (IDV) The proposed IDV approach trained on ID (chest X-ray 14) and OOD data (IRMA and ImageNet) achieved, on average, 0.999 OOD detection AUC across the three data sets.
• Score: 58.720142291102135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To investigate the impact of OOD radiographs on existing chest X-ray classification models and to increase their robustness against OOD data. The study employed the commonly used chest X-ray classification model, CheXnet, trained on the chest X-ray 14 data set, and tested its robustness against OOD data using three public radiography data sets: IRMA, Bone Age, and MURA, and the ImageNet data set. To detect OOD data for multi-label classification, we proposed in-distribution voting (IDV). The OOD detection performance is measured across data sets using the area under the receiver operating characteristic curve (AUC) analysis and compared with Mahalanobis-based OOD detection, MaxLogit, MaxEnergy and self-supervised OOD detection (SS OOD). Without additional OOD detection, the chest X-ray classifier failed to discard any OOD images, with an AUC of 0.5. The proposed IDV approach trained on ID (chest X-ray 14) and OOD data (IRMA and ImageNet) achieved, on average, 0.999 OOD AUC across the three data sets, surpassing all other OOD detection methods. Mahalanobis-based OOD detection achieved an average OOD detection AUC of 0.982. IDV trained solely with a few thousand ImageNet images had an AUC 0.913, which was higher than MaxLogit (0.726), MaxEnergy (0.724), and SS OOD (0.476). The performance of all tested OOD detection methods did not translate well to radiography data sets, except Mahalanobis-based OOD detection and the proposed IDV method. Training solely on ID data led to incorrect classification of OOD images as ID, resulting in increased false positive rates. IDV substantially improved the model's ID classification performance, even when trained with data that will not occur in the intended use case or test set, without additional inference overhead.

Related papers

• Evaluating Reliability in Medical DNNs: A Critical Analysis of Feature and Confidence-Based OOD Detection [2.9049649065453336]
  OOD detection methods can be categorised as confidence-based (using the model's output layer for OOD detection) or feature-based (not using the output layer) We show that OOD artefacts can boost a model's softmax confidence in its predictions, due to correlations in training data among other factors. We also show that feature-based methods typically perform worse at distinguishing between inputs that lead to correct and incorrect predictions.
  arXiv  Detail & Related papers  (2024-08-30T15:02:22Z)
• Out-of-distribution Detection with Implicit Outlier Transformation [72.73711947366377]
  Outlier exposure (OE) is powerful in out-of-distribution (OOD) detection. We propose a novel OE-based approach that makes the model perform well for unseen OOD situations.
  arXiv  Detail & Related papers  (2023-03-09T04:36:38Z)
• Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective [55.45202687256175]
  Out-of-distribution (OOD) detection methods assume that they have test ground truths, i.e., whether individual test samples are in-distribution (IND) or OOD. In this paper, we are the first to introduce the unsupervised evaluation problem in OOD detection. We propose three methods to compute Gscore as an unsupervised indicator of OOD detection performance.
  arXiv  Detail & Related papers  (2023-02-16T13:34:35Z)
• Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need [52.88953913542445]
  We find surprisingly that simply using reconstruction-based methods could boost the performance of OOD detection significantly. We take Masked Image Modeling as a pretext task for our OOD detection framework (MOOD)
  arXiv  Detail & Related papers  (2023-02-06T08:24:41Z)
• PnPOOD : Out-Of-Distribution Detection for Text Classification via Plug andPlay Data Augmentation [25.276900899887192]
  We present OOD, a data augmentation technique to perform OOD detection via out-of-domain sample generation. Our method generates high quality discriminative samples close to the class boundaries, resulting in accurate OOD detection at test time. We highlight an important data leakage issue with datasets used in prior attempts at OOD detection and share results on a new dataset for OOD detection that does not suffer from the same problem.
  arXiv  Detail & Related papers  (2021-10-31T14:02:26Z)
• CVAD: A generic medical anomaly detector based on Cascade VAE [2.647674705784439]
  We focus on the generalizability of OOD detection for medical images and propose a self-supervised Cascade Variational autoencoder-based Anomaly Detector (CVAD) We use a variational autoencoders' cascade architecture, which combines latent representation at multiple scales, before being fed to a discriminator to distinguish the OOD data from the in-distribution (ID) data. We compare the performance with the state-of-the-art deep learning models to demonstrate our model's efficacy on various open-access medical imaging datasets for both intra- and inter-class OOD.
  arXiv  Detail & Related papers  (2021-10-29T14:20:43Z)
• Provably Robust Detection of Out-of-distribution Data (almost) for free [124.14121487542613]
  Deep neural networks are known to produce highly overconfident predictions on out-of-distribution (OOD) data. In this paper we propose a novel method where from first principles we combine a certifiable OOD detector with a standard classifier into an OOD aware classifier. In this way we achieve the best of two worlds: certifiably adversarially robust OOD detection, even for OOD samples close to the in-distribution, without loss in prediction accuracy and close to state-of-the-art OOD detection performance for non-manipulated OOD data.
  arXiv  Detail & Related papers  (2021-06-08T11:40:49Z)
• ATOM: Robustifying Out-of-distribution Detection Using Outlier Mining [51.19164318924997]
  Adrial Training with informative Outlier Mining improves robustness of OOD detection. ATOM achieves state-of-the-art performance under a broad family of classic and adversarial OOD evaluation tasks.
  arXiv  Detail & Related papers  (2020-06-26T20:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.