Synthetic but Not Realistic: The Evaluation Challenge in Generative Modelling for Structured Electronic Medical Records

• URL: http://arxiv.org/abs/2606.08903v1
• Date: Mon, 08 Jun 2026 01:07:37 GMT
• Title: Synthetic but Not Realistic: The Evaluation Challenge in Generative Modelling for Structured Electronic Medical Records
• Abstract summary: We introduce a multi-dimensional evaluation framework grounded in epidemiology, assessing descriptive fidelity, clinical utility, and structural validity.<n>We evaluate four representative generative paradigms using PRIME-CVD, a 50,000-person cohort with known ground-truth structure.
• Score: 0.7537475180985093
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthetic healthcare data are widely proposed as privacy-preserving substitutes for real patient data, yet their evaluation remains dominated by statistical similarity and predictive performance that do not reflect clinical validity. We introduce a multi-dimensional evaluation framework grounded in epidemiology, assessing descriptive fidelity, clinical utility, and structural validity, corresponding to descriptive, predictive, and causal questions. We evaluate four representative generative paradigms - GAN-based, VAE-boosted, diffusion-based, and masked modelling - using PRIME-CVD, a 50,000-person cohort with known ground-truth structure. While all models reproduce marginal distributions, none simultaneously preserve subgroup structure, effect estimates, and dependency structure. Notably, models with strong distributional fidelity can exhibit poor calibration and distorted relationships, leading to unreliable inference. These results show that current evaluation practices can overestimate synthetic data quality and motivate domain-informed assessment based on the ability to support valid clinical and scientific conclusions.

Related papers

• The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models [0.1274452325287335]
  We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines.<n>PFMs consistently outperform CNNs in both robustness and domain generalisation, yet model scaling shows diminishing returns.<n> NR-Kfold analysis reveals systematic accuracy loss and increased variability when training-test similarity is broken.
  arXiv  Detail & Related papers  (2026-07-05T16:49:09Z)
• Uncertainty-Aware Foundation Models for Clinical Data [12.07343307130062]
  We propose a framework for uncertainty aware foundation modeling that represents each patient not as a point embedding, but as a distribution over plausible latent states.<n>We integrate this formulation with multimodal encoders and scalable self supervised objectives, combining reconstruction, contrastive alignment, and distributional regularization.
  arXiv  Detail & Related papers  (2026-04-05T16:44:13Z)
• Cross-Linguistic Persona-Driven Data Synthesis for Robust Multimodal Cognitive Decline Detection [20.599682298329213]
  We introduce SynCog, a novel framework integrating controllable zero-shot multimodal data synthesis with Chain-of-Thought deduction fine-tuning.<n>This generative paradigm enables the rapid, zero-shot expansion of clinical corpora across diverse languages.<n>Experiments on the ADReSS and ADReSSo benchmarks demonstrate that augmenting limited clinical data with synthetic phenotypes yields competitive diagnostic performance.
  arXiv  Detail & Related papers  (2026-02-08T14:10:05Z)
• Investigating the Impact of Histopathological Foundation Models on Regressive Prediction of Homologous Recombination Deficiency [52.50039435394964]
  We systematically evaluate foundation models for regression-based tasks.<n>We extract patch-level features from whole slide images (WSI) using five state-of-the-art foundation models.<n>Models are trained to predict continuous HRD scores based on these extracted features across breast, endometrial, and lung cancer cohorts.
  arXiv  Detail & Related papers  (2026-01-29T14:06:50Z)
• Intervention Efficiency and Perturbation Validation Framework: Capacity-Aware and Robust Clinical Model Selection under the Rashomon Effect [8.16102315566872]
  coexistence of multiple models with comparable performance poses fundamental challenges for trustworthy deployment and evaluation.<n>We propose two complementary tools for robust model assessment and selection: Intervention Efficiency (IE) and the Perturbation Validation Framework (PVF)<n>IE is a capacity-aware metric that quantifies how efficiently a model identifies actionable true positives when only limited interventions are feasible.<n>PVF introduces a structured approach to assess the stability of models under data perturbations, identifying models whose performance remains most invariant across noisy or shifted validation sets.
  arXiv  Detail & Related papers  (2025-11-18T10:21:07Z)
• Improving the Generation and Evaluation of Synthetic Data for Downstream Medical Causal Inference [89.5628648718851]
  Causal inference is essential for developing and evaluating medical interventions.<n>Real-world medical datasets are often difficult to access due to regulatory barriers.<n>We present STEAM: a novel method for generating Synthetic data for Treatment Effect Analysis in Medicine.
  arXiv  Detail & Related papers  (2025-10-21T16:16:00Z)
• Bridging the Generalisation Gap: Synthetic Data Generation for Multi-Site Clinical Model Validation [0.3362278589492841]
  Existing model evaluation approaches often rely on real-world datasets, which are limited in availability, embed confounding biases, and lack flexibility needed for systematic experimentation.<n>We propose a novel structured synthetic data framework designed for the controlled robustness of benchmarking model, fairness, and generalisability.
  arXiv  Detail & Related papers  (2025-04-29T11:04:28Z)
• TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network [54.332862955411656]
  Clinical trials are critical for drug development but often suffer from expensive and inefficient patient recruitment. In recent years, machine learning models have been proposed for speeding up patient recruitment via automatically matching patients with clinical trials. We introduce a dynamic tree-based memory network model named TREEMENT to provide accurate and interpretable patient trial matching.
  arXiv  Detail & Related papers  (2023-07-19T12:35:09Z)
• Causal Inference via Nonlinear Variable Decorrelation for Healthcare Applications [60.26261850082012]
  We introduce a novel method with a variable decorrelation regularizer to handle both linear and nonlinear confounding. We employ association rules as new representations using association rule mining based on the original features to increase model interpretability.
  arXiv  Detail & Related papers  (2022-09-29T17:44:14Z)
• Benchmarking Heterogeneous Treatment Effect Models through the Lens of Interpretability [82.29775890542967]
  Estimating personalized effects of treatments is a complex, yet pervasive problem. Recent developments in the machine learning literature on heterogeneous treatment effect estimation gave rise to many sophisticated, but opaque, tools. We use post-hoc feature importance methods to identify features that influence the model's predictions.
  arXiv  Detail & Related papers  (2022-06-16T17:59:05Z)
• Decomposed Adversarial Learned Inference [118.27187231452852]
  We propose a novel approach, Decomposed Adversarial Learned Inference (DALI) DALI explicitly matches prior and conditional distributions in both data and code spaces. We validate the effectiveness of DALI on the MNIST, CIFAR-10, and CelebA datasets.
  arXiv  Detail & Related papers  (2020-04-21T20:00:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.