A Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology

• URL: http://arxiv.org/abs/2608.24688v1
• Date: Tue, 25 Aug 2026 15:17:12 GMT
• Title: A Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology
• Abstract summary: We introduce the oFM, a foundation model developed on a real-world oncology cohort of 1.67 million cancer patients.<n>The oFM encodes daily clinical and molecular episodes and integrates them over time to produce a patient state embedding.<n>In prognostic benchmarks, the oFM improved AUC for treatment response, progression-free survival, and overall survival.
• Score: 2.4534049320113227
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Precision oncology necessitates a longitudinal model of patient state that captures cancer evolution and treatment over time, integrating multimodal observations. We introduce the oFM, a foundation model developed on a real-world oncology cohort of 1.67 million cancer patients that integrates clinical trajectories with DNA, RNA, and H&E pathology. Patient-level partitions were reserved for training, validation, and testing, with over one million patients used for training. The oFM encodes daily clinical and molecular episodes and, along with pathology images, integrates them over time to produce a patient state embedding. We evaluate frozen oFM embeddings against expert-curated clinical and molecular baseline features. In prognostic benchmarks, the oFM improved AUC for treatment response, progression-free survival, and overall survival (0.774 vs. 0.563 for overall survival). Across 11 comparative-treatment cohorts, the oFM embeddings achieved a three-fold higher pooled and scale-normalized treatment-benefit AUTOC than baseline features with improved benefit ranking in 9 of 11 cohorts, and provided stronger prognostic discrimination within both treatment arms. We also evaluated a mechanism discovery framework that interprets downstream models built on oFM embeddings by linking their predicted outcomes to clinically and biologically grounded mechanisms through an evidence-grounded temporal graph, enabling evaluation in clinical and drug-development applications.

Related papers

• A Vision-Language Foundation Model for Precise and Comprehensive Brain Tumor Diagnosis from Preoperative Multimodal Data [50.52606847963128]
  We developed BrainVLM to classify all 12 World Health Organization (WHO) 2021 brain tumor types.<n>BrainVLM was trained on multi-modal data (MRI scans, demographics, and radiology reports) from 40,043 individuals.<n>It was validated on 5,211 patients with pathologically confirmed brain tumors.
  arXiv  Detail & Related papers  (2026-09-15T03:46:03Z)
• Linking spatial biology and clinical histology via Haiku [35.723756132708026]
  We present Haiku, a tri-modal contrastive learning model trained on multiplexed immunofluorescence (mIF)<n>It comprises 26.7 million spatial patches from 3,218 tissue sections across 1,606 patients spanning 11 organ types.
  arXiv  Detail & Related papers  (2026-04-30T18:53:32Z)
• Multimodal Fusion of Histopathology Images and Electronic Health Records for Early Breast Cancer Diagnosis [15.70912899013153]
  This paper presents a systematic framework that integrates patch-level histopathology features with structured clinical data.<n>We train and evaluate unimodal image models (a simple CNN baseline and ResNet-18 with transfer learning) and intermediate-fusion models (XGBoost and a multilayer perceptron)<n>Our results demonstrate that multimodal integration delivers meaningful improvements in both predictive performance and clinical transparency.
  arXiv  Detail & Related papers  (2026-04-18T19:46:34Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Investigating the Impact of Histopathological Foundation Models on Regressive Prediction of Homologous Recombination Deficiency [52.50039435394964]
  We systematically evaluate foundation models for regression-based tasks.<n>We extract patch-level features from whole slide images (WSI) using five state-of-the-art foundation models.<n>Models are trained to predict continuous HRD scores based on these extracted features across breast, endometrial, and lung cancer cohorts.
  arXiv  Detail & Related papers  (2026-01-29T14:06:50Z)
• Chronic Kidney Disease Prognosis Prediction Using Transformer [2.054117570146147]
  Chronic Kidney Disease (CKD) affects nearly 10% of the global population and often progresses to end-stage renal failure.<n>We present a transformer-based framework for predicting CKD progression using multi-modal electronic health records.
  arXiv  Detail & Related papers  (2025-11-04T07:52:17Z)
• A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer [54.58205672910646]
  RenalCLIP is a visual-language foundation model for characterization, diagnosis and prognosis of renal mass.<n>It achieved better performance and superior generalizability across 10 core tasks spanning the full clinical workflow of kidney cancer.
  arXiv  Detail & Related papers  (2025-08-22T17:48:19Z)
• Development of Interactive Nomograms for Predicting Short-Term Survival in ICU Patients with Aplastic Anemia [3.5626691568652507]
  Aplastic anemia is a rare, life-threatening hematologic disorder characterized by pancytopenia and bone marrow failure.<n>We used the MIMIC-IV database to identify ICU patients with aplastic anemia and extracted clinical features from five domains.<n> Logistic regression and Cox regression models were constructed to predict 7-, 14-, and 28-day mortality.
  arXiv  Detail & Related papers  (2025-05-23T23:01:11Z)
• Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning [0.8428580558402998]
  We developed a novel artificial neural network architecture, multimodal transformer-based simple temporal attention (MMTSimTA) network.<n>We integrated pre- and on-treatment blood measurements, prescribed medications and CT-based volumes of organs from a large pan-cancer cohort of 694 patients treated with immunotherapy to predict mortality.<n>Our findings show that integrating noninvasive longitudinal data using our novel architecture yields an improved multimodal prognostic performance, especially in short-term survival prediction.
  arXiv  Detail & Related papers  (2024-11-27T11:44:06Z)
• TopoTxR: A topology-guided deep convolutional network for breast parenchyma learning on DCE-MRIs [49.69047720285225]
  We propose a novel topological approach that explicitly extracts multi-scale topological structures to better approximate breast parenchymal structures. We empirically validate emphTopoTxR using the VICTRE phantom breast dataset. Our qualitative and quantitative analyses suggest differential topological behavior of breast tissue in treatment-na"ive imaging.
  arXiv  Detail & Related papers  (2024-11-05T19:35:10Z)
• Improved Prognostic Prediction of Pancreatic Cancer Using Multi-Phase CT by Integrating Neural Distance and Texture-Aware Transformer [37.55853672333369]
  This paper proposes a novel learnable neural distance that describes the precise relationship between the tumor and vessels in CT images of different patients. The developed risk marker was the strongest predictor of overall survival among preoperative factors.
  arXiv  Detail & Related papers  (2023-08-01T12:46:02Z)
• MIA-Prognosis: A Deep Learning Framework to Predict Therapy Response [58.0291320452122]
  This paper aims at a unified deep learning approach to predict patient prognosis and therapy response. We formalize the prognosis modeling as a multi-modal asynchronous time series classification task. Our predictive model could further stratify low-risk and high-risk patients in terms of long-term survival.
  arXiv  Detail & Related papers  (2020-10-08T15:30:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.