Beyond Accuracy: An Empirical Study of Uncertainty Estimation in Imputation

• URL: http://arxiv.org/abs/2511.21607v1
• Date: Wed, 26 Nov 2025 17:27:59 GMT
• Title: Beyond Accuracy: An Empirical Study of Uncertainty Estimation in Imputation
• Authors: Zarin Tahia Hossain, Mostafa Milani,
• Abstract summary: imputation methods differ in how they represent and quantify uncertainty.<n>Uncertainty is estimated through three complementary routes: multi-run variability, conditional sampling, and predictive-distribution modeling.<n>We analyze method-specific trade-offs among accuracy calibration, runtime, identify stable configurations, and offer guidelines for selecting uncertainty-aware imputers.
• Score: 0.5524804393257919
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Handling missing data is a central challenge in data-driven analysis. Modern imputation methods not only aim for accurate reconstruction but also differ in how they represent and quantify uncertainty. Yet, the reliability and calibration of these uncertainty estimates remain poorly understood. This paper presents a systematic empirical study of uncertainty in imputation, comparing representative methods from three major families: statistical (MICE, SoftImpute), distribution alignment (OT-Impute), and deep generative (GAIN, MIWAE, TabCSDI). Experiments span multiple datasets, missingness mechanisms (MCAR, MAR, MNAR), and missingness rates. Uncertainty is estimated through three complementary routes: multi-run variability, conditional sampling, and predictive-distribution modeling, and evaluated using calibration curves and the Expected Calibration Error (ECE). Results show that accuracy and calibration are often misaligned: models with high reconstruction accuracy do not necessarily yield reliable uncertainty. We analyze method-specific trade-offs among accuracy, calibration, and runtime, identify stable configurations, and offer guidelines for selecting uncertainty-aware imputers in data cleaning and downstream machine learning pipelines.

Related papers

• Uncertainty Weighted Gradients for Model Calibration [22.39558434131574]
  Deep networks often produce over-confident or under-confident predictions, leading to miscalibration.<n>We propose a unified loss framework for focal loss and its variants, where we mainly attribute their superiority in model calibration to the loss weighting factor.<n>Our method achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2025-03-26T04:16:05Z)
• Enhancing accuracy of uncertainty estimation in appearance-based gaze tracking with probabilistic evaluation and calibration [13.564919425738163]
  Uncertainty in appearance-based gaze tracking is critical for ensuring reliable downstream applications.<n>Current uncertainty-aware approaches adopt probabilistic models to acquire uncertainties by following distributions in the training dataset.<n>We propose a correction strategy based on probability calibration to mitigate biases in the estimated uncertainties of the trained models.
  arXiv  Detail & Related papers  (2025-01-24T19:33:55Z)
• Probabilistic Modeling of Disparity Uncertainty for Robust and Efficient Stereo Matching [61.73532883992135]
  We propose a new uncertainty-aware stereo matching framework.<n>We adopt Bayes risk as the measurement of uncertainty and use it to separately estimate data and model uncertainty.
  arXiv  Detail & Related papers  (2024-12-24T23:28:20Z)
• Towards Understanding Variants of Invariant Risk Minimization through the Lens of Calibration [0.6906005491572401]
  We show that Information Bottleneck-based IRM achieves consistent calibration across different environments. Our empirical evidence indicates that models exhibiting consistent calibration across environments are also well-calibrated.
  arXiv  Detail & Related papers  (2024-01-31T02:08:43Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Towards Reliable Medical Image Segmentation by Modeling Evidential Calibrated Uncertainty [57.023423137202485]
  Concerns regarding the reliability of medical image segmentation persist among clinicians.<n>We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks.<n>By leveraging subjective logic theory, we explicitly model probability and uncertainty for medical image segmentation.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)
• Reliability-Aware Prediction via Uncertainty Learning for Person Image Retrieval [51.83967175585896]
  UAL aims at providing reliability-aware predictions by considering data uncertainty and model uncertainty simultaneously. Data uncertainty captures the noise" inherent in the sample, while model uncertainty depicts the model's confidence in the sample's prediction.
  arXiv  Detail & Related papers  (2022-10-24T17:53:20Z)
• Approaching Neural Network Uncertainty Realism [53.308409014122816]
  Quantifying or at least upper-bounding uncertainties is vital for safety-critical systems such as autonomous vehicles. We evaluate uncertainty realism -- a strict quality criterion -- with a Mahalanobis distance-based statistical test. We adopt it to the automotive domain and show that it significantly improves uncertainty realism compared to a plain encoder-decoder model.
  arXiv  Detail & Related papers  (2021-01-08T11:56:12Z)
• Improving model calibration with accuracy versus uncertainty optimization [17.056768055368384]
  A well-calibrated model should be accurate when it is certain about its prediction and indicate high uncertainty when it is likely to be inaccurate. We propose an optimization method that leverages the relationship between accuracy and uncertainty as an anchor for uncertainty calibration. We demonstrate our approach with mean-field variational inference and compare with state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-12-14T20:19:21Z)
• Calibration of Model Uncertainty for Dropout Variational Inference [1.8065361710947976]
  In this paper, different logit scaling methods are extended to dropout variational inference to recalibrate model uncertainty. Experimental results show that logit scaling considerably reduce miscalibration by means of UCE.
  arXiv  Detail & Related papers  (2020-06-20T14:12:55Z)
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically [4.552831400384914]
  Calibrated uncertainty estimates in machine learning are crucial to many fields such as autonomous vehicles, medicine, and weather and climate forecasting. We present a calibration method for regression settings that does not assume a particular uncertainty distribution over the error: Calibrating Regression Uncertainty Distributions Empirically (CRUDE) CRUDE demonstrates consistently sharper, better calibrated, and more accurate uncertainty estimates than state-of-the-art techniques.
  arXiv  Detail & Related papers  (2020-05-26T03:08:43Z)
• Learning to Predict Error for MRI Reconstruction [67.76632988696943]
  We demonstrate that predictive uncertainty estimated by the current methods does not highly correlate with prediction error. We propose a novel method that estimates the target labels and magnitude of the prediction error in two steps.
  arXiv  Detail & Related papers  (2020-02-13T15:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.