A multimodal and temporal foundation model for virtual patient representations at healthcare system scale

• URL: http://arxiv.org/abs/2604.18570v2
• Date: Tue, 21 Apr 2026 21:55:35 GMT
• Title: A multimodal and temporal foundation model for virtual patient representations at healthcare system scale
• Abstract summary: We introduce Apollo, a multimodal temporal foundation model trained and evaluated on over three decades of longitudinal hospital records.<n>Apollo learns a unified representation space integrating over 100 thousand unique medical events in our clinical vocabulary.<n>We show that model predictions align with clinically-interpretable multimodal biomarkers.
• Score: 36.24986458898116
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Modern medicine generates vast multimodal data across siloed systems, yet no existing model integrates the full breadth and temporal depth of the clinical record into a unified patient representation. We introduce Apollo, a multimodal temporal foundation model trained and evaluated on over three decades of longitudinal hospital records from a major US hospital system, composed of 25 billion records from 7.2 million patients, representing 28 distinct medical modalities and 12 major medical specialties. Apollo learns a unified representation space integrating over 100 thousand unique medical events in our clinical vocabulary as well as images and clinical text. This "atlas of medical concepts" forms a computational substrate for modeling entire patient care journeys comprised of sequences of structured and unstructured events, which are compressed by Apollo into virtual patient representations. To assess the potential of these whole-patient representations, we created 322 prognosis and retrieval tasks from a held-out test set of 1.4 million patients. We demonstrate the generalized clinical forecasting potential of Apollo embeddings, including predicting new disease onset risk up to five years in advance (95 tasks), disease progression (78 tasks), treatment response (59 tasks), risk of treatment-related adverse events (17 tasks), and hospital operations endpoints (12 tasks). Using feature attribution techniques, we show that model predictions align with clinically-interpretable multimodal biomarkers. We evaluate semantic similarity search on 61 retrieval tasks, and moreover demonstrate the potential of Apollo as a multimodal medical search engine using text and image queries. Together, these modeling capabilities establish the foundation for computable medicine, where the full context of patient care becomes accessible to computational reasoning.

Related papers

• NOAH: Learning the Full Patient Journey. A Longitudinal Multimodal Time-Aware Model for Representation and Forecasting [15.183085955033148]
  We introduce NOAH, a time-aware, task-agnostic, generative transformer model representing and forecasting the full patient journey.<n>NOAH processes medical images, time-series and numeric signals, categorical events, as well as structured and unstructured clinical records.<n>It generates highly informative and predictive patient state representations that demonstrate strong performance in probing for clinical outcomes.
  arXiv  Detail & Related papers  (2026-09-08T17:56:13Z)
• Medical world models in healthcare: foundations, applications, and challenges for trustworthy clinical translation [9.298791886645303]
  Medical world models offer a framework for extending medical artificial intelligence beyond static prediction.<n>This Review defines the conceptual boundaries, technical foundations, application domains, and evidence requirements of the field.
  arXiv  Detail & Related papers  (2026-07-28T03:35:47Z)
• Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases [71.12461204050985]
  MedSP1000 is an SP-derived interactive benchmark for clinical-agent evaluation.<n>It converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubrics.<n>Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios.
  arXiv  Detail & Related papers  (2026-06-03T17:17:16Z)
• Simulating clinical interventions with a generative multimodal model of human physiology [49.828582089394445]
  HealthFormer is a decoder-only transformer that models the human physiological trajectory generatively.<n>We tokenise each participant's health trajectory across 667 measurements spanning seven domains.<n>HealthFormer improves prediction for 27 of 30 incident-disease and mortality endpoints, exceeding established clinical risk scores in every comparison.
  arXiv  Detail & Related papers  (2026-04-30T14:10:30Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Generative Medical Event Models Improve with Scale [11.04430289614517]
  We present the largest scaling-law study of medical event data.<n>We pretrained a series of compute-optimal models with up to 1 billion parameters.<n>We studied 78 real-world tasks, including diagnosis prediction, disease prognosis, and healthcare operations.
  arXiv  Detail & Related papers  (2025-08-16T17:00:51Z)
• Foresight -- Deep Generative Modelling of Patient Timelines using Electronic Health Records [46.024501445093755]
  Temporal modelling of medical history can be used to forecast and simulate future events, estimate risk, suggest alternative diagnoses or forecast complications. We present Foresight, a novel GPT3-based pipeline that uses NER+L tools (i.e. MedCAT) to convert document text into structured, coded concepts.
  arXiv  Detail & Related papers  (2022-12-13T19:06:00Z)
• Modelling Patient Trajectories Using Multimodal Information [0.0]
  We propose a solution to model patient trajectories that combines different types of information and considers the temporal aspect of clinical data. The developed solution was evaluated on two different clinical outcomes, unexpected patient readmission and disease progression.
  arXiv  Detail & Related papers  (2022-09-09T10:20:54Z)
• Integrated multimodal artificial intelligence framework for healthcare applications [3.6222901399459215]
  We propose and evaluate a unified Holistic AI in Medicine framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations.
  arXiv  Detail & Related papers  (2022-02-25T22:08:09Z)
• Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence [79.038671794961]
  We launch the Unified CT-COVID AI Diagnostic Initiative (UCADI), where the AI model can be distributedly trained and independently executed at each host institution. Our study is based on 9,573 chest computed tomography scans (CTs) from 3,336 patients collected from 23 hospitals located in China and the UK.
  arXiv  Detail & Related papers  (2021-11-18T00:43:41Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.