Uncertainty Identifies Difficult Samples Across Methods: A Multi-Task Study on a Heterogeneous Skin Lesion Dataset

• URL: http://arxiv.org/abs/2608.14768v1
• Date: Fri, 14 Aug 2026 14:14:13 GMT
• Title: Uncertainty Identifies Difficult Samples Across Methods: A Multi-Task Study on a Heterogeneous Skin Lesion Dataset
• Abstract summary: Five UQ methods are compared on accuracy, calibration, uncertainty decomposition, and risk-coverage.<n>Even methods with narrow entropy rank the same samples as hard.<n>The choice of method matters more for calibration and uncertainty decomposition, where Deep Ensembles is the clear winner.
• Score: 8.271195591114806
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Skin lesion classifiers can be confidently wrong on the cases that matter most, so knowing when a prediction should not be trusted is clinically as useful as the prediction. We study uncertainty quantification on a dataset pooled from many ISIC sources, with a shared backbone and two jointly learned heads: a binary malignant versus non-malignant head and a five-class diagnostic head. Five UQ methods (MC Dropout, DropConnect, Flipout, Deep Ensembles, DUQ) are compared on accuracy, calibration, uncertainty decomposition, and risk-coverage. Difficulty is largely method-agnostic: even methods with narrow entropy distributions rank the same samples as hard (per-sample entropy correlations of $0.54$ to $0.91$). The choice of method matters more for calibration and uncertainty decomposition, where Deep Ensembles is the clear winner, than for finding difficult cases. The ranking is also good enough that deferring the most uncertain cases removes a disproportionate share of errors, supporting uncertainty-based selective referral, evaluated here in-distribution only.

Related papers

• Heckman-Corrected Epistemic Uncertainty: Selection on Unobservables Defeats Importance Weighting [0.0]
  Training data for machine learning is routinely collected by a selection process the model never sees.<n>Etonians solved the harder case in 1979: Heckman's two-equation model fits a probit selection equation and an outcome equation linked through correlated errors.<n>The Heckman correction restores coverage when the selection equation has an instrument -- a variable affecting selection but not the outcome -- and degrades measurably without one.
  arXiv  Detail & Related papers  (2026-07-07T03:59:46Z)
• EnTrust: Modeling Inter-Modal Conflict for Trustworthy Multimodal Medical Image Analysis [41.67522319355107]
  EnTrust is a framework that treats inter-modal conflict as the primary source of predictive uncertainty.<n>It achieves state-of-the-art segmentation accuracy while reducing calibration error by 40% compared to the strongest baseline.
  arXiv  Detail & Related papers  (2026-06-19T12:43:50Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• Multidimensional Uncertainty Quantification via Optimal Transport [87.97146725546502]
  We take a multidimensional view on uncertainty quantification (UQ) by stacking complementary UQ measures into a vector.<n>VecUQ-OT shows high efficiency even when individual measures fail.
  arXiv  Detail & Related papers  (2025-09-26T14:09:03Z)
• Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression [102.24287051757469]
  We study self-supervised covariance estimation in deep heteroscedastic regression.<n>We derive an upper bound on the 2-Wasserstein distance between normal distributions.<n>Experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.
  arXiv  Detail & Related papers  (2025-02-14T22:37:11Z)
• Measuring Uncertainty Disentanglement Error in Classification [7.687215328455751]
  We present a set of experiments to benchmark the quality of disentanglement using multiple datasets.<n>We show that Deep Ensembles with Information Theoretic disentanglement have the best Disentanglement Error.
  arXiv  Detail & Related papers  (2024-08-22T07:42:43Z)
• Structural-Based Uncertainty in Deep Learning Across Anatomical Scales: Analysis in White Matter Lesion Segmentation [8.64414399041931]
  Uncertainty quantification (UQ) is an indicator of the trustworthiness of automated deep-learning (DL) tools in the context of white matter lesion (WML) segmentation. We develop measures for quantifying uncertainty at lesion and patient scales, derived from structural prediction discrepancies. The results from a multi-centric MRI dataset of 444 patients demonstrate that our proposed measures more effectively capture model errors at the lesion and patient scales.
  arXiv  Detail & Related papers  (2023-11-15T13:04:57Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• Towards Reliable Medical Image Segmentation by Modeling Evidential Calibrated Uncertainty [57.023423137202485]
  Concerns regarding the reliability of medical image segmentation persist among clinicians.<n>We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks.<n>By leveraging subjective logic theory, we explicitly model probability and uncertainty for medical image segmentation.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)
• Multi-Label Quantification [78.83284164605473]
  Quantification, variously called "labelled prevalence estimation" or "learning to quantify", is the supervised learning task of generating predictors of the relative frequencies of the classes of interest in unsupervised data samples. We propose methods for inferring estimators of class prevalence values that strive to leverage the dependencies among the classes of interest in order to predict their relative frequencies more accurately.
  arXiv  Detail & Related papers  (2022-11-15T11:29:59Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• Identifying Incorrect Classifications with Balanced Uncertainty [21.130311978327196]
  Uncertainty estimation is critical for cost-sensitive deep-learning applications. We propose the distributional imbalance to model the imbalance in uncertainty estimation as two kinds of distribution biases. We then propose Balanced True Class Probability framework, which learns an uncertainty estimator with a novel Distributional Focal Loss objective.
  arXiv  Detail & Related papers  (2021-10-15T11:52:31Z)
• Recalibration of Aleatoric and Epistemic Regression Uncertainty in Medical Imaging [2.126171264016785]
  Well-calibrated uncertainty in regression allows robust rejection of unreliable predictions or detection of out-of-distribution samples. $ sigma $ scaling is able to reliably recalibrate predictive uncertainty.
  arXiv  Detail & Related papers  (2021-04-26T07:18:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.