Rethinking external validation for the target population: Capturing patient-level similarity with a generative model

• URL: http://arxiv.org/abs/2605.11284v1
• Date: Mon, 11 May 2026 22:11:03 GMT
• Title: Rethinking external validation for the target population: Capturing patient-level similarity with a generative model
• Abstract summary: We propose a framework that quantifies each external patient's similarity to the development data.<n>We use generative models, specifically autoencoders, to estimate similarity.<n>Results: Our framework revealed substantial variation in model performance across similarity-defined subgroups.
• Score: 1.5535982620458004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: External validation is essential for assessing the transportability of predictive models. However, its interpretation is often confounded by differences between external and development populations. This study introduces a framework to distinguish model deficiencies from case-mix effects. Method: We propose a framework that quantifies each external patient's similarity to the development data and measures performance in subgroups with varying levels of alignment to the development distribution. We use generative models, specifically autoencoders, to estimate similarity, offering a more flexible alternative to traditional linear approaches and enabling validation without sharing the original development data. The utility of autoencoder-based similarity measure is demonstrated using synthetic data, and the framework's application is illustrated using data from the Netherlands Heart Registration (NHR) to predict mortality after transcatheter aortic valve implantation. Results: Our framework revealed substantial variation in model performance across similarity-defined subgroups, differences that remain hidden under conventional external validation yet can meaningfully alter conclusions. In several settings, conventional external validation suggested poor overall performance. However, after accounting for differences in patient characteristics, for some sub-groups, the model performance was consistent with internal validation results. Conversely, apparently acceptable overall performance could mask clinically relevant performance deficits in specific subgroups. Conclusion: The proposed framework enhances the interpretability of external validation by linking model performance to population alignment with the development data. This provides a more principled basis for deciding whether a model is transportable and to which patients it can be safely applied.

Related papers

• A Statistical Framework for Data-Driven Discovery of Differential Performance in Clinical Risk Prediction Models [0.0]
  Predictive models employing artificial intelligence (AI) and machine learning (ML) are increasingly being used for decision support in healthcare settings.<n>These models may exhibit differential performance across population subgroups defined by race, age, sex, and other factors.<n>In real-world settings, important model performance disparities may arise in unknown subgroups defined by multiple intersecting characteristics.
  arXiv  Detail & Related papers  (2026-08-02T15:53:22Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• SeFEF: A Seizure Forecasting Evaluation Framework [0.0]
  We introduce a Python-based framework aimed at streamlining the development, assessment, and documentation of seizure forecasting algorithms.<n>The framework automates data labeling, cross-validation splitting, forecast post-processing, performance evaluation, and reporting.<n>It supports various forecasting horizons and includes a model card that documents implementation details, training and evaluation settings, and performance metrics.
  arXiv  Detail & Related papers  (2025-10-13T11:10:27Z)
• Bridging the Generalisation Gap: Synthetic Data Generation for Multi-Site Clinical Model Validation [0.3362278589492841]
  Existing model evaluation approaches often rely on real-world datasets, which are limited in availability, embed confounding biases, and lack flexibility needed for systematic experimentation.<n>We propose a novel structured synthetic data framework designed for the controlled robustness of benchmarking model, fairness, and generalisability.
  arXiv  Detail & Related papers  (2025-04-29T11:04:28Z)
• Influence Functions for Scalable Data Attribution in Diffusion Models [52.92223039302037]
  Diffusion models have led to significant advancements in generative modelling.<n>Yet their widespread adoption poses challenges regarding data attribution and interpretability.<n>We develop an influence functions framework to address these challenges.
  arXiv  Detail & Related papers  (2024-10-17T17:59:02Z)
• Flexible Nonparametric Inference for Causal Effects under the Front-Door Model [2.6900047294457683]
  We develop novel one-step and targeted minimum loss-based estimators for both the average treatment effect and the average treatment effect on the treated under front-door assumptions.<n>Our estimators are built on multiple parameterizations of the observed data distribution, including approaches that avoid mediator density entirely.<n>We show how these constraints can be leveraged to improve the efficiency of causal effect estimators.
  arXiv  Detail & Related papers  (2023-12-15T22:04:53Z)
• A Causal Inference Framework for Leveraging External Controls in Hybrid Trials [1.7942265700058988]
  We consider the challenges associated with causal inference in settings where data from a randomized trial is augmented with control data from an external source. We propose estimators, review efficiency bounds, and an approach for efficient doubly-robust estimation. We apply the framework to a trial investigating the effect of risdisplam on motor function in patients with spinal muscular atrophy.
  arXiv  Detail & Related papers  (2023-05-15T19:15:32Z)
• Bayesian Networks for the robust and unbiased prediction of depression and its symptoms utilizing speech and multimodal data [65.28160163774274]
  We apply a Bayesian framework to capture the relationships between depression, depression symptoms, and features derived from speech, facial expression and cognitive game data collected at thymia.
  arXiv  Detail & Related papers  (2022-11-09T14:48:13Z)
• Causal Inference via Nonlinear Variable Decorrelation for Healthcare Applications [60.26261850082012]
  We introduce a novel method with a variable decorrelation regularizer to handle both linear and nonlinear confounding. We employ association rules as new representations using association rule mining based on the original features to increase model interpretability.
  arXiv  Detail & Related papers  (2022-09-29T17:44:14Z)
• Selecting Treatment Effects Models for Domain Adaptation Using Causal Knowledge [82.5462771088607]
  We propose a novel model selection metric specifically designed for ITE methods under the unsupervised domain adaptation setting. In particular, we propose selecting models whose predictions of interventions' effects satisfy known causal structures in the target domain.
  arXiv  Detail & Related papers  (2021-02-11T21:03:14Z)
• Estimating the Effects of Continuous-valued Interventions using Generative Adversarial Networks [103.14809802212535]
  We build on the generative adversarial networks (GANs) framework to address the problem of estimating the effect of continuous-valued interventions. Our model, SCIGAN, is flexible and capable of simultaneously estimating counterfactual outcomes for several different continuous interventions. To address the challenges presented by shifting to continuous interventions, we propose a novel architecture for our discriminator.
  arXiv  Detail & Related papers  (2020-02-27T18:46:21Z)
• NestedVAE: Isolating Common Factors via Weak Supervision [45.366986365879505]
  We identify the connection between the task of bias reduction and that of isolating factors common between domains. To isolate the common factors we combine the theory of deep latent variable models with information bottleneck theory. Two outer VAEs with shared weights attempt to reconstruct the input and infer a latent space, whilst a nested VAE attempts to reconstruct the latent representation of one image, from the latent representation of its paired image.
  arXiv  Detail & Related papers  (2020-02-26T15:49:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.