Foundation Models for Clinical Records at Health System Scale

• URL: http://arxiv.org/abs/2507.00574v1
• Date: Tue, 01 Jul 2025 08:52:33 GMT
• Title: Foundation Models for Clinical Records at Health System Scale
• Abstract summary: We present a novel generative pretraining strategy for sequential EHR data using next-visit event prediction.<n>Our model learns to autoregressively generate various tokenized clinical events for the next visit based on patient history.
• Score: 40.88151645546234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale pretraining has transformed modeling of language and other data types, but its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present a novel generative pretraining strategy for sequential EHR data using next-visit event prediction. Our model learns to autoregressively generate various tokenized clinical events for the next visit based on patient history and inherently handles the joint prediction of heterogeneous data types. Additionally, we introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Our model is evaluated via zero-shot prediction for forecasting dementia and knee osteoarthritis incidence within 2 and 5 years, and the model performance rivals a fully fine-tuned masked pretrained Transformer baseline, demonstrating that our approach captures complex clinical dependencies without requiring costly task-specific fine-tuning.

Related papers

• EHR-RAGp: Retrieval-Augmented Prototype-Guided Foundation Model for Electronic Health Records [0.9989630065856642]
  EHR contain rich longitudinal patient information and are widely used in predictive modeling applications.<n>We introduce EHR-RAGp, a retrieval-augmented foundation model that dynamically integrates the most relevant patient history across diverse clinical event types.
  arXiv  Detail & Related papers  (2026-05-12T16:17:03Z)
• Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction [34.2525943791427]
  We present RAVEN, a novel generative pretraining strategy for sequential EHR data based on Recurrence-Aware next-Visit EveNt prediction.<n>Our model learns to autoregressively generate tokenized clinical events for the next visit conditioned on patient history.
  arXiv  Detail & Related papers  (2026-03-25T17:42:47Z)
• One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models [12.630229861635476]
  We propose ORA, a time-to-event pretraining objective that jointly models event timing and associated measurements.<n>Our results suggest a broader takeaway: pretraining objectives that account for EHR structure are critical for expanding downstream capabilities and generalizability.
  arXiv  Detail & Related papers  (2026-01-31T06:15:46Z)
• The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHR [3.9541492005870604]
  We introduce SMB-Structure, a world model for structured EHR that grounds a joint-embedding prediction architecture (JEPA) with next-token prediction (SFT)<n>We validate across two large-scale cohorts: Memorial Sloan Kettering (23,319 oncology patients; 323,000+ patient-years) and INSPECT (19,402 pulmonary embolism patients)
  arXiv  Detail & Related papers  (2026-01-29T18:49:37Z)
• Investigating the Impact of Histopathological Foundation Models on Regressive Prediction of Homologous Recombination Deficiency [52.50039435394964]
  We systematically evaluate foundation models for regression-based tasks.<n>We extract patch-level features from whole slide images (WSI) using five state-of-the-art foundation models.<n>Models are trained to predict continuous HRD scores based on these extracted features across breast, endometrial, and lung cancer cohorts.
  arXiv  Detail & Related papers  (2026-01-29T14:06:50Z)
• Analysis of heart failure patient trajectories using sequence modeling [2.5101181751229915]
  Mamba architecture outperformed an advanced Transformer (Transformer++) based on Llama in handling long context lengths.<n>Llama achieves the highest predictive discrimination, best calibration, and showed robustness across all tasks, followed by Mambas.<n>This paper presents a first ablation study with systematic design choices for input tokenization, model configuration and temporal data preprocessing.
  arXiv  Detail & Related papers  (2025-11-20T22:43:25Z)
• Building the EHR Foundation Model via Next Event Prediction [5.378917071184147]
  Next Event Prediction (NEP) is a framework that enhances Large Language Models' temporal reasoning.<n>NEP explicitly models disease progression patterns and causal relationships.<n>Our analyses reveal dual benefits: state-of-the-art prediction accuracy combined with clinically interpretable attention patterns.
  arXiv  Detail & Related papers  (2025-09-29T23:27:51Z)
• Zero-shot Medical Event Prediction Using a Generative Pre-trained Transformer on Electronic Health Records [8.575985305475355]
  We show that a foundation model trained on EHRs can perform predictive tasks in a zero-shot manner.<n>Unlike supervised approaches requiring extensive labeled data, our method enables the model to forecast a next medical event purely from a pretraining knowledge.
  arXiv  Detail & Related papers  (2025-03-07T19:26:47Z)
• Deep State-Space Generative Model For Correlated Time-to-Event Predictions [54.3637600983898]
  We propose a deep latent state-space generative model to capture the interactions among different types of correlated clinical events. Our method also uncovers meaningful insights about the latent correlations among mortality and different types of organ failures.
  arXiv  Detail & Related papers  (2024-07-28T02:42:36Z)
• Recent Advances in Predictive Modeling with Electronic Health Records [71.19967863320647]
  utilizing EHR data for predictive modeling presents several challenges due to its unique characteristics. Deep learning has demonstrated its superiority in various applications, including healthcare.
  arXiv  Detail & Related papers  (2024-02-02T00:31:01Z)
• IGNITE: Individualized GeNeration of Imputations in Time-series Electronic health records [6.630372114304835]
  We propose a novel deep-learning model that learns the underlying patient dynamics to generate personalized values conditioning on an individual's demographic characteristics and treatments.<n>Our proposed model, IGNITE, utilise a conditional dual-variational autoencoder augmented with dual-stage attention to generate missing values for an individual.<n>We show that IGNITE outperforms state-of-the-art approaches in missing data reconstruction and task prediction.
  arXiv  Detail & Related papers  (2024-01-09T07:57:21Z)
• MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation [58.93221876843639]
  This paper introduces a novel, end-to-end diffusion-based risk prediction model, named MedDiffusion. It enhances risk prediction performance by creating synthetic patient data during training to enlarge sample space. It discerns hidden relationships between patient visits using a step-wise attention mechanism, enabling the model to automatically retain the most vital information for generating high-quality data.
  arXiv  Detail & Related papers  (2023-10-04T01:36:30Z)
• Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment [72.50906475214457]
  The goal of sequential event prediction is to estimate the next event based on a sequence of historical events. In practice, the next-event prediction models are trained with sequential data collected at one time. We propose a framework with hierarchical branching structures for learning context-specific representations.
  arXiv  Detail & Related papers  (2022-10-24T07:54:13Z)
• SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models [48.07469930813923]
  This work aims to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. We introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.
  arXiv  Detail & Related papers  (2021-08-31T08:23:56Z)
• Pre-training transformer-based framework on large-scale pediatric claims data for downstream population-specific tasks [3.1580072841682734]
  This study presents the Claim Pre-Training (Claim-PT) framework, a generic pre-training model that first trains on the entire pediatric claims dataset. The effective knowledge transfer is completed through the task-aware fine-tuning stage. We conducted experiments on a real-world claims dataset with more than one million patient records.
  arXiv  Detail & Related papers  (2021-06-24T15:25:41Z)
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data [62.29031007761901]
  This paper proposes a novel contrastive regularized clinical classification model. We introduce two unique positive sampling strategies specifically tailored for EHR data. Our framework yields highly competitive experimental results in predicting the mortality risk on real-world COVID-19 EHR data.
  arXiv  Detail & Related papers  (2021-04-07T06:02:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.