Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification

• URL: http://arxiv.org/abs/2508.04457v1
• Date: Wed, 06 Aug 2025 13:58:17 GMT
• Title: Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification
• Authors: Simon Baur, Wojciech Samek, Jackie Ma,
• Abstract summary: We evaluate 13 uncertainty quantification methods for convolutional (ResNet) and transformer-based (Vision Transformer) architectures.<n>We extend Evidential Deep Learning, HetClass NNs, and Deep Deterministic Uncertainty to the multi-label setting.
• Score: 11.21639536740362
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reliable uncertainty quantification is crucial for trustworthy decision-making and the deployment of AI models in medical imaging. While prior work has explored the ability of neural networks to quantify predictive, epistemic, and aleatoric uncertainties using an information-theoretical approach in synthetic or well defined data settings like natural image classification, its applicability to real life medical diagnosis tasks remains underexplored. In this study, we provide an extensive uncertainty quantification benchmark for multi-label chest X-ray classification using the MIMIC-CXR-JPG dataset. We evaluate 13 uncertainty quantification methods for convolutional (ResNet) and transformer-based (Vision Transformer) architectures across a wide range of tasks. Additionally, we extend Evidential Deep Learning, HetClass NNs, and Deep Deterministic Uncertainty to the multi-label setting. Our analysis provides insights into uncertainty estimation effectiveness and the ability to disentangle epistemic and aleatoric uncertainties, revealing method- and architecture-specific strengths and limitations.

Related papers

• Hesitation is defeat? Connecting Linguistic and Predictive Uncertainty [2.8186733524862158]
  This paper investigates the relationship between predictive uncertainty and human/linguistic uncertainty, as estimated from free-text reports labelled by rule-based labellers.<n>The results demonstrate good model performance, but also a modest correlation between predictive and linguistic uncertainty, highlighting the challenges in aligning machine uncertainty with human interpretation.
  arXiv  Detail & Related papers  (2025-05-06T18:34:37Z)
• Unified Uncertainty Estimation for Cognitive Diagnosis Models [70.46998436898205]
  We propose a unified uncertainty estimation approach for a wide range of cognitive diagnosis models. We decompose the uncertainty of diagnostic parameters into data aspect and model aspect. Our method is effective and can provide useful insights into the uncertainty of cognitive diagnosis.
  arXiv  Detail & Related papers  (2024-03-09T13:48:20Z)
• Empirical Validation of Conformal Prediction for Trustworthy Skin Lesions Classification [3.7305040207339286]
  We develop Conformal Prediction, Monte Carlo Dropout, and Evidential Deep Learning approaches to assess uncertainty quantification in deep neural networks. Results: The experimental results demonstrate a significant enhancement in uncertainty quantification with the utilization of the Conformal Prediction method. Our conclusion highlights a robust and consistent performance of conformal prediction across diverse testing conditions.
  arXiv  Detail & Related papers  (2023-12-12T17:37:16Z)
• Uncertainty Quantification in Machine Learning Based Segmentation: A Post-Hoc Approach for Left Ventricle Volume Estimation in MRI [0.0]
  Left ventricular (LV) volume estimation is critical for valid diagnosis and management of various cardiovascular conditions.<n>Recent machine learning advancements, particularly U-Net-like convolutional networks, have facilitated automated segmentation for medical images.<n>This study proposes a novel methodology for post-hoc uncertainty estimation in LV volume prediction.
  arXiv  Detail & Related papers  (2023-10-30T13:44:55Z)
• Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentation [7.313010190714819]
  quantifying uncertainty associated with model predictions is crucial in critical clinical applications. Deep learning based methods for automatic organ segmentation have shown promise in aiding diagnosis and treatment planning. It is unclear which method is preferred in the medical image analysis setting.
  arXiv  Detail & Related papers  (2023-08-15T00:09:33Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Towards Reliable Medical Image Segmentation by utilizing Evidential Calibrated Uncertainty [52.03490691733464]
  We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks. By leveraging subjective logic theory, we explicitly model probability and uncertainty for the problem of medical image segmentation. DeviS incorporates an uncertainty-aware filtering module, which utilizes the metric of uncertainty-calibrated error to filter reliable data.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)
• Uncertainty in Extreme Multi-label Classification [81.14232824864787]
  eXtreme Multi-label Classification (XMC) is an essential task in the era of big data for web-scale machine learning applications. In this paper, we aim to investigate general uncertainty quantification approaches for tree-based XMC models with a probabilistic ensemble-based framework. In particular, we analyze label-level and instance-level uncertainty in XMC, and propose a general approximation framework based on beam search to efficiently estimate the uncertainty with a theoretical guarantee under long-tail XMC predictions.
  arXiv  Detail & Related papers  (2022-10-18T20:54:33Z)
• Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets [0.7734726150561088]
  A lack of statistically rigorous uncertainty quantification is a significant factor undermining trust in AI results. Recent developments in distribution-free uncertainty quantification present practical solutions for these issues. We demonstrate a technique for forming ordinal prediction sets that are guaranteed to contain the correct stenosis severity.
  arXiv  Detail & Related papers  (2022-07-05T18:01:20Z)
• The Unreasonable Effectiveness of Deep Evidential Regression [72.30888739450343]
  A new approach with uncertainty-aware regression-based neural networks (NNs) shows promise over traditional deterministic methods and typical Bayesian NNs. We detail the theoretical shortcomings and analyze the performance on synthetic and real-world data sets, showing that Deep Evidential Regression is a quantification rather than an exact uncertainty.
  arXiv  Detail & Related papers  (2022-05-20T10:10:32Z)
• Bayesian Uncertainty Estimation of Learned Variational MRI Reconstruction [63.202627467245584]
  We introduce a Bayesian variational framework to quantify the model-immanent (epistemic) uncertainty. We demonstrate that our approach yields competitive results for undersampled MRI reconstruction.
  arXiv  Detail & Related papers  (2021-02-12T18:08:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.