Pretraining EHR Foundation Models with Patient-Aware Sampling

• URL: http://arxiv.org/abs/2607.22114v1
• Date: Fri, 24 Jul 2026 09:10:43 GMT
• Title: Pretraining EHR Foundation Models with Patient-Aware Sampling
• Abstract summary: Patient Sampling is a pretraining sequence-construction method that allows us to control how training signal is distributed across patients.<n>We show that Patient Sampling with controllable weighting improves performance on real-world EHR data.<n>These results identify training and validation as important and underexplored design choices for autoregressive EHR foundation models.
• Score: 15.261305009306978
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias. We propose Patient Sampling, a pretraining sequence-construction method that allows us to control how training signal is distributed across patients. We compare this method to the standard approach, which we refer to as Global Stream. We show that stochastic Patient Sampling with controllable weighting improves performance on real-world EHR data. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.

Related papers

• Improving Patient Subtyping on Longitudinal Data using Representations from Mamba-based Architecture [5.577596968571274]
  We propose a self-supervised Mamba-based model that learns effective EHR representations and enables enhanced patient subtyping.<n>We evaluate the proposed model on public and private real-world EHR datasets to classify the data.<n>Our findings offer valuable insights into subtyping patients based on temporal records from EHR models.
  arXiv  Detail & Related papers  (2026-06-26T21:45:07Z)
• FRESH: Information-Geometric Calibration of Patient-Level Models to Aggregate Evidence [0.0]
  Fresh is a method for incorporating population-level summary results into predictive models trained on patient-level data.<n>The method produces patient-level predictions from a re-calibrated model that matches a set of specified aggregate statistics for a target population.
  arXiv  Detail & Related papers  (2026-05-15T17:50:11Z)
• Integrating Genomics into Multimodal EHR Foundation Models [56.31910745104141]
  This paper introduces an innovative EHR foundation model that integrates Polygenic Risk Scores (PRS) as a foundational data modality.<n>The framework aims to learn complex relationships between clinical data and genetic predispositions.<n>This approach is pivotal for unlocking new insights into disease prediction, proactive health management, risk stratification, and personalized treatment strategies.
  arXiv  Detail & Related papers  (2025-10-24T15:56:40Z)
• Cross-Representation Benchmarking in Time-Series Electronic Health Records for Clinical Outcome Prediction [44.23284500920266]
  This benchmark standardises data curation and evaluation across two distinct clinical settings.<n>Experiments reveal that event stream models consistently deliver the strongest performance.<n>We find that feature selection strategies must be adapted to the clinical setting.
  arXiv  Detail & Related papers  (2025-10-10T09:03:47Z)
• Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models [69.06149482021071]
  We propose a novel EHR data generation model called EHRPD. It is a diffusion-based model designed to predict the next visit based on the current one while also incorporating time interval estimation. We conduct experiments on two public datasets and evaluate EHRPD from fidelity, privacy, and utility perspectives.
  arXiv  Detail & Related papers  (2024-06-20T02:20:23Z)
• MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation [58.93221876843639]
  This paper introduces a novel, end-to-end diffusion-based risk prediction model, named MedDiffusion. It enhances risk prediction performance by creating synthetic patient data during training to enlarge sample space. It discerns hidden relationships between patient visits using a step-wise attention mechanism, enabling the model to automatically retain the most vital information for generating high-quality data.
  arXiv  Detail & Related papers  (2023-10-04T01:36:30Z)
• Federated Learning of Medical Concepts Embedding using BEHRT [0.0]
  We propose a federated learning approach for learning medical concepts embedding. Our approach is based on embedding model like BEHRT, a deep neural sequence model for EHR. We compare the performance of a model trained with FL against a model trained on centralized data.
  arXiv  Detail & Related papers  (2023-05-22T14:05:39Z)
• SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models [48.07469930813923]
  This work aims to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. We introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.
  arXiv  Detail & Related papers  (2021-08-31T08:23:56Z)
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data [62.29031007761901]
  This paper proposes a novel contrastive regularized clinical classification model. We introduce two unique positive sampling strategies specifically tailored for EHR data. Our framework yields highly competitive experimental results in predicting the mortality risk on real-world COVID-19 EHR data.
  arXiv  Detail & Related papers  (2021-04-07T06:02:04Z)
• Adversarial Sample Enhanced Domain Adaptation: A Case Study on Predictive Modeling with Electronic Health Records [57.75125067744978]
  We propose a data augmentation method to facilitate domain adaptation. adversarially generated samples are used during domain adaptation. Results confirm the effectiveness of our method and the generality on different tasks.
  arXiv  Detail & Related papers  (2021-01-13T03:20:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.