Calibration, Uncertainty Communication, and Deployment Readiness in CKD Risk Prediction: A Framework Evaluation Study

• URL: http://arxiv.org/abs/2605.21566v1
• Date: Wed, 20 May 2026 17:12:48 GMT
• Title: Calibration, Uncertainty Communication, and Deployment Readiness in CKD Risk Prediction: A Framework Evaluation Study
• Abstract summary: Machine learning models for chronic kidney disease often post strong discrimination scores on internal test sets.<n>We trained five classifiers on the UCI CKD dataset (400 patients, 62.5% CKD prevalence)<n>We evaluated each across calibration quality, conformal prediction coverage, and an eight-criterion deployment readiness framework.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning models for chronic kidney disease (CKD) risk prediction often post strong discrimination scores on internal test sets. Calibration and uncertainty quantification get far less attention, leaving clinicians without reliable information about whether the probability outputs are accurate. We trained five classifiers on the UCI CKD dataset (400 patients, 62.5% CKD prevalence): logistic regression, random forest, XGBoost, SVM with Platt scaling, and Gaussian naive Bayes. We evaluated each across calibration quality, conformal prediction coverage, and an eight-criterion deployment readiness framework. A distributional stress-test applied the best-calibrated variant of each model to the open-access MIMIC-IV demo cohort (97 patients, 23.7% CKD) to assess behaviour under prevalence shift and feature missingness. We measured calibration before and after Platt scaling and isotonic regression using Expected Calibration Error and Brier Score, and quantified uncertainty through split conformal prediction targeting 90% marginal coverage. All five models reached AUROC 1.00 on the UCI test set. Isotonic recalibration reduced internal ECE to 0.000-0.022. On MIMIC-IV, AUROC fell to 0.48-0.58, ECE rose to 0.68-0.76, and conformal coverage dropped from 0.80-0.98 to 0.21-0.25 against a 90% target. No model scored above 4 out of 16 on the deployment readiness checklist. Near-perfect internal performance did not transfer. Calibration stability and conformal coverage should be evaluated on external data before any clinical prediction model moves toward deployment.

Related papers

• Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models [0.0]
  Screening models routinely report areas under receiver operating characteristic curve (AUROC) near 0.89.<n>We benchmarked ten classifiers spanning linear, tree-ensemble, neural, glass-box, and tabular foundation classes for prevalent myocardial infarction.<n>Removing two post-diagnostic features cost every model 0.049-0.051 AUROC, collapsing the field into a 0.0045-wide band.<n>One threshold detected 75.4% of women's infarctions against 89.0% of men's; editing the model's shape functions reduced the gap to 0.010.
  arXiv  Detail & Related papers  (2026-09-10T17:29:15Z)
• Reliability-Aware Hybrid-K Ensemble Selection for Cervical Cytology Classification: Integrating Discrimination, Calibration, and Selective Prediction [3.093890460224435]
  This study proposes a reliability-aware Hybrid-K ensemble selection framework for multiclass cervical classification.<n>Nine deep learning architectures were evaluated using a fixed stratified five-fold partition and three training seeds.<n>The final Hybrid-2 ensemble, comprising Swin-Tiny and TinyViT-5M, reduced AURC by 43%, NLL by 17%, and WC-ECE by 36% relative to the best individual model.
  arXiv  Detail & Related papers  (2026-09-01T02:30:41Z)
• A Domain-Structured Ensemble Framework for Perioperative Outcome Prediction Using Electronic Health Record Data [0.0]
  Perioperative risk prediction models are often limited by narrow surgical populations, incomplete data, and limited interpretability.<n>We present a domain-structured ensemble framework for outcome prediction using routinely collected electronic health record (EHR) data.<n>We demonstrate the framework using postoperative delirium in a case-control sample of 5,386 surgical encounters from a statewide health information exchange.
  arXiv  Detail & Related papers  (2026-08-09T21:15:12Z)
• Predictive Entropy Links Calibration and Paraphrase Sensitivity in Medical Vision-Language Models [2.064612766965483]
  We show five uncertainty quantification methods on MedGemma 4BIT across in distribution MIMIC CXR and outof distribution PadChest chest X ray datasets, with cross architecture validation on LLaVA RAD7B.<n>For well calibrated single model methods, predictive entropy from one forward pass predicts which samples will flip under rephrasing AUROC 0.711 on MedGemma, 0.878 on LLaVARAD p 10 4, enabling a single entropy threshold to flag both unreliable and sensitive predictions.
  arXiv  Detail & Related papers  (2026-04-10T04:18:47Z)
• Tipping the Balance: Impact of Class Imbalance Correction on the Performance of Clinical Risk Prediction Models [2.2534253247996214]
  Class-imbalance correction techniques are commonly applied to improve model performance in settings with rare outcomes.<n>This study evaluated the effect of widely used resampling strategies on both discrimination and calibration across real-world clinical prediction tasks.
  arXiv  Detail & Related papers  (2026-02-27T12:29:16Z)
• EMBC Special Issue: Calibrated Uncertainty for Trustworthy Clinical Gait Analysis Using Probabilistic Multiview Markerless Motion Capture [1.260797434681533]
  Video-based human movement analysis holds potential for movement assessment in clinical practice and research.<n>This study evaluates the calibration and reliability of a probabilistic multi-view markerless motion capture method.<n>Probability model reconstruction quantifies uncertainty, allowing it to identify unreliable outputs without the need for concurrent ground-truth instrumentation.
  arXiv  Detail & Related papers  (2026-01-29T23:50:19Z)
• Explainable Admission-Level Predictive Modeling for Prolonged Hospital Stay in Elderly Populations: Challenges in Low- and Middle-Income Countries [65.4286079244589]
  Prolonged length of stay (pLoS) is a significant factor associated with the risk of adverse in-hospital events.<n>We develop and explain a predictive model for pLos using admission-level patient and hospital administrative data.
  arXiv  Detail & Related papers  (2026-01-07T23:35:24Z)
• Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems [0.29465623430708904]
  Uncalibrated scores can invert preferences, naive confidence intervals on uncalibrated scores achieve near-0% coverage, and importance-weighted estimators collapse under limited overlap.<n>We introduce Causal Judge Evaluation, a framework that fixes all three failures.
  arXiv  Detail & Related papers  (2025-12-11T22:16:24Z)
• A Critical Perspective on Finite Sample Conformal Prediction Theory in Medical Applications [46.14968220063657]
  Conformal prediction (CP) is a popular tool that allows users to turn uncertainty estimates into uncertainty estimates with statistical guarantees.<n>We show that although statistical guarantees hold for calibration sets of arbitrary size, the practical utility of these guarantees does highly depend on the size of the calibration set.<n>This observation is relevant in medical domains because data is often scarce and obtaining large calibration sets is therefore infeasible.
  arXiv  Detail & Related papers  (2025-12-09T16:59:31Z)
• An Explainable and Fair AI Tool for PCOS Risk Assessment: Calibration, Subgroup Equity, and Interactive Clinical Deployment [0.10026496861838446]
  This paper presents a fairness-audited and interpretable machine learning framework for predicting polycystic ovary syndrome (PCOS)<n>The framework integrated SHAP-based feature attributions with demographic audits to connect predictive explanations with observed disparities for actionable insights.<n>A Streamlit-based web interface enables real-time PCOS risk assessment, Rotterdam criteria evaluation, and interactive 'what-if' analysis.
  arXiv  Detail & Related papers  (2025-11-08T16:14:56Z)
• Conformal Lesion Segmentation for 3D Medical Images [82.92159832699583]
  We propose a risk-constrained framework that calibrates data-driven thresholds via conformalization to ensure the test-time FNR remains below a target tolerance.<n>We validate the statistical soundness and predictive performance of CLS on six 3D-LS datasets across five backbone models, and conclude with actionable insights for deploying risk-aware segmentation in clinical practice.
  arXiv  Detail & Related papers  (2025-10-19T08:21:00Z)
• Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
  Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.<n>Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.<n>We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
  arXiv  Detail & Related papers  (2024-10-13T15:37:11Z)
• Towards Reliable Medical Image Segmentation by Modeling Evidential Calibrated Uncertainty [57.023423137202485]
  Concerns regarding the reliability of medical image segmentation persist among clinicians.<n>We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks.<n>By leveraging subjective logic theory, we explicitly model probability and uncertainty for medical image segmentation.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data [81.00385374948125]
  We present UNcertaInTy-based hEalth risk prediction (UNITE) model. UNITE provides accurate disease risk prediction and uncertainty estimation leveraging multi-sourced health data. We evaluate UNITE on real-world disease risk prediction tasks: nonalcoholic fatty liver disease (NASH) and Alzheimer's disease (AD) UNITE achieves up to 0.841 in F1 score for AD detection, up to 0.609 in PR-AUC for NASH detection, and outperforms various state-of-the-art baselines by up to $19%$ over the best baseline.
  arXiv  Detail & Related papers  (2020-10-22T02:28:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.