MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology

• URL: http://arxiv.org/abs/2608.16972v1
• Date: Mon, 17 Aug 2026 12:44:38 GMT
• Title: MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology
• Abstract summary: We propose MultiSigBERT, a unified framework for sequential survival modeling in oncology.<n>Here, narrative medical reports are converted into sentence geometric embeddings by extracting and averaging contextual word embeddings.<n>These representations are then compressed via modality-specific PCA and geometric embeddings to form joint temporal trajectories.<n>The computed Signature features are finally incorporated as high dimensional features into a LASSO-regularized Cox model to estimate individualized risk scores.
• Score: 1.4526244723917887
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning has become an essential component of modern healthcare, where the integration of heterogeneous data sources offers unprecedented opportunities to improve clinical decision-making. Electronic Health Records (EHR) contain complementary information -- including narrative clinical reports, numerical measurements, and structured variables -- yet most survival models remain limited to a single modality or fail to exploit the temporal nature of patient trajectories. We propose MultiSigBERT, a unified framework for multimodal sequential survival modeling in oncology based on path signature representations. Here, narrative medical reports (free-text) are converted into sentence embeddings by extracting and averaging contextual word embeddings. These representations are then compressed via modality-specific PCA and concatenated with structured covariates to form joint temporal trajectories which are then encoded using the Signature transform, a tool from Rough Paths theory that efficiently captures higher-order temporal interactions across modalities without supervision needed. The computed Signature features are finally incorporated as high dimensional features into a LASSO-regularized Cox model to estimate individualized risk scores. The performance of our novel MultiSigBERT pipeline is illustrated on the analysis of a real-world oncology cohort from the Léon Bérard Center, comprising over 120,000 medical reports and structured records from more than 2,500 patients. The model achieves a concordance index of 0.743 (sd 0.029) on an independent test set, demonstrating the benefit of jointly modeling multimodal temporal dynamics together with patient-level geometric structure for survival prediction.

Related papers

• ChronoSurv: A Clinical Pathway-Guided Graph Framework for Multimodal Survival Analysis [0.0]
  ChronoSurv represents patient care as a progression-aware clinical trajectory using directed graphs aligned with key diagnostic steps.<n> Experimental results on two public datasets demonstrate that ChronoSurv achieves state-of-the-art discriminative performance.
  arXiv  Detail & Related papers  (2026-06-17T14:46:47Z)
• Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion [1.4203297056262238]
  We propose a novel deep learning framework for patient-level survival prediction.<n>It integrates whole-slide histology features, RNA-seq expression profiles, and clinical variables.<n>Our architecture combines an ABMIL moduleciteilse2018attention for slide-level representation with feedforward encoders for RNA and clinical data.
  arXiv  Detail & Related papers  (2026-05-12T13:09:25Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Counterfactual Understanding via Retrieval-aware Multimodal Modeling for Time-to-Event Survival Prediction [1.5713805841057418]
  CURE is a framework that advances counterfactual survival modeling via comprehensive multimodal embedding and latent retrieval.<n> Experimental results on METABRIC and TCGA-LUAD datasets demonstrate that proposed CURE model consistently outperforms strong baselines in survival analysis.
  arXiv  Detail & Related papers  (2026-02-23T15:53:25Z)
• Integrating Genomics into Multimodal EHR Foundation Models [56.31910745104141]
  This paper introduces an innovative EHR foundation model that integrates Polygenic Risk Scores (PRS) as a foundational data modality.<n>The framework aims to learn complex relationships between clinical data and genetic predispositions.<n>This approach is pivotal for unlocking new insights into disease prediction, proactive health management, risk stratification, and personalized treatment strategies.
  arXiv  Detail & Related papers  (2025-10-24T15:56:40Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• SigBERT: Combining Narrative Medical Reports and Rough Path Signature Theory for Survival Risk Estimation in Oncology [1.5425688173297465]
  SigBERT is an innovative temporal survival analysis framework designed to process a large number of clinical reports per patient.<n>It processes timestamped medical reports by extracting and averaging word embeddings into sentence embeddings.<n>It was trained and evaluated on a real-world oncology dataset from the L'eon B'erard Center corpus, with a C-index score of 0.75 (sd 0.014) on the independent test cohort.
  arXiv  Detail & Related papers  (2025-07-25T12:33:25Z)
• Deep Survival Analysis in Multimodal Medical Data: A Parametric and Probabilistic Approach with Competing Risks [47.19194118883552]
  We introduce a multimodal deep learning framework for survival analysis capable of modeling both single and competing risks scenarios.<n>We propose SAMVAE (Survival Analysis Multimodal Variational Autoencoder), a novel deep learning architecture designed for survival prediction.
  arXiv  Detail & Related papers  (2025-07-10T14:29:48Z)
• Adaptable Cardiovascular Disease Risk Prediction from Heterogeneous Data using Large Language Models [70.64969663547703]
  AdaCVD is an adaptable CVD risk prediction framework built on large language models extensively fine-tuned on over half a million participants from the UK Biobank.<n>It addresses key clinical challenges across three dimensions: it flexibly incorporates comprehensive yet variable patient information; it seamlessly integrates both structured data and unstructured text; and it rapidly adapts to new patient populations using minimal additional data.
  arXiv  Detail & Related papers  (2025-05-30T14:42:02Z)
• Continually Evolved Multimodal Foundation Models for Cancer Prognosis [50.43145292874533]
  Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates.<n>Previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information.<n>Existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals.<n>Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities.
  arXiv  Detail & Related papers  (2025-01-30T06:49:57Z)
• Survival Prediction in Lung Cancer through Multi-Modal Representation Learning [9.403446155541346]
  This paper presents a novel approach to survival prediction by harnessing comprehensive information from CT and PET scans, along with associated Genomic data. We aim to develop a robust predictive model for survival outcomes by integrating multi-modal imaging data with genetic information.
  arXiv  Detail & Related papers  (2024-09-30T10:42:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.