Calibrating Ensembles for Scalable Uncertainty Quantification in Deep Learning-based Medical Segmentation

• URL: http://arxiv.org/abs/2209.09563v1
• Date: Tue, 20 Sep 2022 09:09:48 GMT
• Title: Calibrating Ensembles for Scalable Uncertainty Quantification in Deep Learning-based Medical Segmentation
• Authors: Thomas Buddenkotte, Lorena Escudero Sanchez, Mireia Crispin-Ortuzar, Ramona Woitek, Cathal McCague, James D. Brenton, Ozan \"Oktem, Evis Sala, Leonardo Rundo
• Abstract summary: Uncertainty quantification in automated image analysis is highly desired in many applications. Current uncertainty quantification approaches do not scale well in high-dimensional real-world problems. We propose a scalable and intuitive framework to calibrate ensembles of deep learning models to produce uncertainty quantification measurements.
• Score: 0.42008820076301906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Uncertainty quantification in automated image analysis is highly desired in many applications. Typically, machine learning models in classification or segmentation are only developed to provide binary answers; however, quantifying the uncertainty of the models can play a critical role for example in active learning or machine human interaction. Uncertainty quantification is especially difficult when using deep learning-based models, which are the state-of-the-art in many imaging applications. The current uncertainty quantification approaches do not scale well in high-dimensional real-world problems. Scalable solutions often rely on classical techniques, such as dropout, during inference or training ensembles of identical models with different random seeds to obtain a posterior distribution. In this paper, we show that these approaches fail to approximate the classification probability. On the contrary, we propose a scalable and intuitive framework to calibrate ensembles of deep learning models to produce uncertainty quantification measurements that approximate the classification probability. On unseen test data, we demonstrate improved calibration, sensitivity (in two out of three cases) and precision when being compared with the standard approaches. We further motivate the usage of our method in active learning, creating pseudo-labels to learn from unlabeled images and human-machine collaboration.

Related papers

• Instance-wise Uncertainty for Class Imbalance in Semantic Segmentation [4.147659576493158]
  State of the art methods increasingly rely on deep learning models, known to incorrectly estimate uncertainty and be overconfident in predictions. This is particularly problematic in semantic segmentation due to inherent class imbalance. A novel training methodology specifically designed for semantic segmentation is presented.
  arXiv  Detail & Related papers  (2024-07-17T14:38:32Z)
• Uncertainty Quantification for cross-subject Motor Imagery classification [5.62479170374811]
  Uncertainty Quantification aims to determine when a Machine Learning model is likely to be wrong. Deep Ensembles performed best, both in terms of classification performance and cross-subject Uncertainty Quantification performance. Standard CNNs with Softmax output performed better than some of the more advanced methods.
  arXiv  Detail & Related papers  (2024-03-14T09:48:48Z)
• Confidence-Aware and Self-Supervised Image Anomaly Localisation [7.099105239108548]
  We discuss an improved self-supervised single-class training strategy that supports the approximation of probabilistic inference with loosen feature locality constraints. Our method is integrated into several out-of-distribution (OOD) detection models and we show evidence that our method outperforms the state-of-the-art on various benchmark datasets.
  arXiv  Detail & Related papers  (2023-03-23T12:48:47Z)
• Multi-Head Multi-Loss Model Calibration [13.841172927454204]
  We introduce a form of simplified ensembling that bypasses the costly training and inference of deep ensembles. Specifically, each head is trained to minimize a weighted Cross-Entropy loss, but the weights are different among the different branches. We show that the resulting averaged predictions can achieve excellent calibration without sacrificing accuracy in two challenging datasets.
  arXiv  Detail & Related papers  (2023-03-02T09:32:32Z)
• Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise [62.997667081978825]
  In high-risk environments, deep learning models need to be able to judge their uncertainty and reject inputs when there is a significant chance of misclassification. We conduct a rigorous evaluation of the most commonly used uncertainty and robustness methods for the classification of Whole Slide Images. We observe that ensembles of methods generally lead to better uncertainty estimates as well as an increased robustness towards domain shifts and label noise.
  arXiv  Detail & Related papers  (2023-01-03T11:34:36Z)
• On Calibrating Semantic Segmentation Models: Analyses and An Algorithm [51.85289816613351]
  We study the problem of semantic segmentation calibration. Model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. We propose a simple, unifying, and effective approach, namely selective scaling.
  arXiv  Detail & Related papers  (2022-12-22T22:05:16Z)
• Post-hoc Uncertainty Learning using a Dirichlet Meta-Model [28.522673618527417]
  We propose a novel Bayesian meta-model to augment pre-trained models with better uncertainty quantification abilities. Our proposed method requires no additional training data and is flexible enough to quantify different uncertainties. We demonstrate our proposed meta-model approach's flexibility and superior empirical performance on these applications.
  arXiv  Detail & Related papers  (2022-12-14T17:34:11Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning [49.04790688256481]
  The goal of generalized zero-shot learning (GZSL) is to recognise both seen and unseen classes. Most GZSL methods typically learn to synthesise visual representations from semantic information on the unseen classes. We propose a novel framework that leverages dual variational autoencoders with a triplet loss to learn discriminative latent features.
  arXiv  Detail & Related papers  (2021-01-09T05:21:27Z)
• Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View [82.80085730891126]
  We provide the first modernally precise analysis of linear multiclass classification. Our analysis reveals that the classification accuracy is highly distribution-dependent. The insights gained may pave the way for a precise understanding of other classification algorithms.
  arXiv  Detail & Related papers  (2020-11-16T05:17:29Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.