Robustifying and Selecting Cohort-Appropriate Prognostic Models under Distributional Shifts

• URL: http://arxiv.org/abs/2604.16537v1
• Date: Thu, 16 Apr 2026 21:10:41 GMT
• Title: Robustifying and Selecting Cohort-Appropriate Prognostic Models under Distributional Shifts
• Abstract summary: External validation is widely regarded as the gold standard for prognostic model evaluation.<n>We tested whether successful external calibration depends largely on similarity in co variables and outcomes between validation cohorts.<n>We propose two strategies to improve transportability of prognostic models across cohorts.
• Score: 3.8031924942083517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: External validation is widely regarded as the gold standard for prognostic model evaluation. In this study, we challenge the assumption that successful external calibration guarantees model generalizability and propose two complementary strategies to improve transportability of prognostic models across cohorts. Using six real-world surgical cohorts from tertiary academic centers, we tested whether successful external calibration depends largely on similarity in covariates and outcomes between training and validation cohorts, quantified using Kullback-Leibler (KL) divergence, with calibration assessed by the Integrated Calibration Index (ICI). From the model-developer's perspective, we trained the "best-on-average" prognostic model by tuning toward a meta-analysis-derived covariate and outcome distribution as an approximation of the broader target population. From the end-user perspective, we proposed a simple measure for cohort outcome similarity to identify, among published models, the one most suitable for a given target cohort in terms of both calibration and clinical utility. External calibration worsened as distributional mismatch increased. Higher KL divergence was associated with higher ICI in both surgery-alone (Spearman $ρ=0.614$, $p=0.004$) and surgery + adjuvant chemotherapy cohorts (Spearman $ρ=0.738$, $p<0.001$). Meta-analysis-informed weighting improved calibration in most settings without materially affecting discrimination, with the clearest benefit when evaluated on the aggregated external population ($p=0.037$). Models developed in more similar cohorts achieved lower ICI in surgery-alone (Spearman $ρ=0.803$, $p<0.001$) and surgery + adjuvant chemotherapy cohorts (Spearman $ρ=0.737$, $p<0.001$), and provided greater clinical utility on DCA.

Related papers

• Evaluator-Dependent Patient-Adaptive ECG Lead-Channel Allocation [3.337151338735509]
  Patient-conditioned acquisition policies for ECG lead-channel selection can outperform population-wide fixed protocols.<n>We study this evaluator dependence empirically on PTB-XL by freezing two policies trained with a controlled arbitrary-mask logistic evaluator.<n>Three post-hoc sensitivity analyses -- common-reference scoring, training the strong evaluator on a mixture of policy-generated and random masks, and evaluator-aligned Strong-MGA policy training -- each preserve a positive interaction interval.
  arXiv  Detail & Related papers  (2026-08-27T09:04:31Z)
• A Domain-Structured Ensemble Framework for Perioperative Outcome Prediction Using Electronic Health Record Data [0.0]
  Perioperative risk prediction models are often limited by narrow surgical populations, incomplete data, and limited interpretability.<n>We present a domain-structured ensemble framework for outcome prediction using routinely collected electronic health record (EHR) data.<n>We demonstrate the framework using postoperative delirium in a case-control sample of 5,386 surgical encounters from a statewide health information exchange.
  arXiv  Detail & Related papers  (2026-08-09T21:15:12Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Efficient Variance-reduced Estimation from Generative EHR Models: The SCOPE and REACH Estimators [1.0765331200875645]
  Generative models trained using tokenized electronic health record timelines show promise for clinical outcome prediction.<n>Existing approaches suffer from three key limitations: sparse estimate distributions that poorly differentiate patient risk levels, extreme computational costs, and high sampling variance.<n>We propose two new estimators: the Sum of Conditional Outcome Probability Estimator (SCOPE) and Risk Estimation from Anticipated Conditional Hazards (REACH), that leverage next-token probability distributions discarded by standard Monte Carlo.
  arXiv  Detail & Related papers  (2026-02-03T16:49:44Z)
• Intersectional Fairness in Vision-Language Models for Medical Image Disease Classification [25.30858592524878]
  Cross-Modal Alignment Consistency (CMAC-MMD) is a training framework that standardises diagnostic certainty across intersectional patient subgroups.<n>In the dermatology cohort, the proposed method reduced the overall intersectional missed diagnosis gap (difference in True Positive Rate, $$TPR) from 0.50 to 0.26.<n>For glaucoma screening, the method reduced $$TPR from 0.41 to 0.31, achieving a better AUC of 0.72 (vs. 0.71 baseline)
  arXiv  Detail & Related papers  (2025-12-17T09:47:29Z)
• Mitigating Calibration Bias Without Fixed Attribute Grouping for Improved Fairness in Medical Imaging Analysis [2.8943928153775826]
  Cluster-Focal to first identify poorly calibrated samples, cluster them into groups, and then introduce group-wise focal loss to improve calibration bias. We evaluate our method on skin lesion classification with the public HAM10000 dataset, and on predicting future lesional activity for multiple sclerosis (MS) patients.
  arXiv  Detail & Related papers  (2023-07-04T14:14:12Z)
• On the explainability of hospitalization prediction on a large COVID-19 patient dataset [45.82374977939355]
  We develop various AI models to predict hospitalization on a large (over 110$k$) cohort of COVID-19 positive-tested US patients. Despite high data unbalance, the models reach average precision 0.96-0.98 (0.75-0.85), recall 0.96-0.98 (0.74-0.85), and $F_score 0.97-0.98 (0.79-0.83) on the non-hospitalized (or hospitalized) class.
  arXiv  Detail & Related papers  (2021-10-28T10:23:38Z)
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data [62.29031007761901]
  This paper proposes a novel contrastive regularized clinical classification model. We introduce two unique positive sampling strategies specifically tailored for EHR data. Our framework yields highly competitive experimental results in predicting the mortality risk on real-world COVID-19 EHR data.
  arXiv  Detail & Related papers  (2021-04-07T06:02:04Z)
• Adversarial Sample Enhanced Domain Adaptation: A Case Study on Predictive Modeling with Electronic Health Records [57.75125067744978]
  We propose a data augmentation method to facilitate domain adaptation. adversarially generated samples are used during domain adaptation. Results confirm the effectiveness of our method and the generality on different tasks.
  arXiv  Detail & Related papers  (2021-01-13T03:20:20Z)
• Bayesian prognostic covariate adjustment [59.75318183140857]
  Historical data about disease outcomes can be integrated into the analysis of clinical trials in many ways. We build on existing literature that uses prognostic scores from a predictive model to increase the efficiency of treatment effect estimates.
  arXiv  Detail & Related papers  (2020-12-24T05:19:03Z)
• Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score [59.75318183140857]
  Estimating causal effects from randomized experiments is central to clinical research. Most methods for historical borrowing achieve reductions in variance by sacrificing strict type-I error rate control.
  arXiv  Detail & Related papers  (2020-12-17T21:10:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.