Autoregressive EHR Foundation Models with Multimodal Inputs

• URL: http://arxiv.org/abs/2607.22264v1
• Date: Fri, 24 Jul 2026 12:59:20 GMT
• Title: Autoregressive EHR Foundation Models with Multimodal Inputs
• Abstract summary: Autoregressive foundation models trained on tokenized electronic health records can support zero-shot clinical prediction.<n>We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes.
• Score: 14.402677557506367
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. We investigate two key design choices: (1) how to compress long per-modality sequences (e.g., ECG time series) before they enter the multi-modal cross-attention. This feature may be essential to reduce compute overheads and may be beneficial for generalization; (2) how the choice of pretrained encoder for each modality impacts downstream performance. Through controlled ablations on MIMIC-IV, we show that the best latent-compression configurations outperforms both uncompressed cross-attention and mean pooling. Encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further show that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline. This implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.

Related papers

• Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning [16.941311033409626]
  We introduce Latent-Attention Masked Autoencoders (LAMAE), a multimodal, structure-aware masked autoencoder.<n>LAMAE exchanges information directly in the latent space through a shared latent-attention module operating over a study-view-entity hierarchy.<n>LAMAE outperforms modality-specific pretraining and strong contrastive and vision-language baselines across multimodal hospital-stay tasks.
  arXiv  Detail & Related papers  (2026-09-10T17:49:44Z)
• Physics-Informed Deep Learning for False Ventricular Tachycardia Alarm Reduction in the ICU [44.313512671851306]
  False ventricular tachycardia (VT) alarms are a leading contributor to alarm fatigue in intensive care units.<n>We propose a deep learning framework combining a 1D SE-ResNet with ICU-realistic data augmentations and a physics-informed auxiliary reconstruction task.
  arXiv  Detail & Related papers  (2026-09-08T16:33:07Z)
• ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding [47.35612571473169]
  EEG visual decoding aims to recover visual semantics from non-invasive neural time-series signals.<n>Existing methods rely on fixed visual or textual anchors whose semantic relations may become misaligned with EEG representations that vary across trials, subjects, and learning stages.<n>We propose Progressive Contrastive Alignment (ProCA), a unified and model-agnostic framework for adaptive neural-semantic alignment.
  arXiv  Detail & Related papers  (2026-09-04T12:48:25Z)
• Foundation Model for Cardiac Time Series via Masked Latent Attention [19.265319330946465]
  Electrocardiograms (ECGs) are among the most widely available clinical signals and play a central role in cardiovascular diagnosis.<n>We introduce the latent attention masked autoencoder (LAMAE) FM that directly exploits this structure by learning cross-lead connection mechanisms during self-supervised pretraining.<n>Our approach models higher-order interactions across leads through latent attention, enabling permutation-invariant aggregation and adaptive weighting of lead-specific representations.
  arXiv  Detail & Related papers  (2026-03-27T14:41:44Z)
• MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models [59.180043227905294]
  We propose MedPruner, a training-free and model-agnostic hierarchical token pruning framework for efficient 3D medical image understanding.<n>We show that MedPruner enables models such as MedGemma to maintain or even exceed their original performance while retaining fewer than 5% of visual tokens.
  arXiv  Detail & Related papers  (2026-03-12T07:37:00Z)
• MedAD-R1: Eliciting Consistent Reasoning in Interpretible Medical Anomaly Detection via Consistency-Reinforced Policy Optimization [46.65200216642429]
  We introduce MedAD-38K, the first large-scale, multi-modal, and multi-center benchmark for MedAD featuring diagnostic Chain-of-Thought (CoT) annotations alongside structured Visual Question-Answering (VQA) pairs.<n>Our proposed model, MedAD-R1, achieves state-of-the-art (SOTA) performance on the MedAD-38K benchmark, outperforming strong baselines by more than 10%.
  arXiv  Detail & Related papers  (2026-02-01T07:56:10Z)
• Leveraging Foundational Models and Simple Fusion for Multi-modal Physiological Signal Analysis [0.0]
  We adapt the CBraMod encoder for large-scale self-supervised ECG pretraining.<n>We utilize a pre-trained CBraMod encoder for EEG and pre-train a symmetric ECG encoder.<n>Our approach achieves near state-of-the-art performance, demonstrating that carefully designed physiological encoders, even with straightforward fusion, substantially improve downstream performance.
  arXiv  Detail & Related papers  (2025-12-17T09:49:06Z)
• EnECG: Efficient Ensemble Learning for Electrocardiogram Multi-task Foundation Model [46.84040404474695]
  EnECG is an ensemble-based framework that integrates multiple specialized foundation models, each excelling in different aspects of ECG interpretation.<n>We show that EnECG can help reduce computational and memory costs while maintaining the strong representational power of foundation models.<n>This framework not only enhances feature extraction and predictive performance but also ensures practical efficiency for real-world clinical applications.
  arXiv  Detail & Related papers  (2025-11-28T07:22:33Z)
• PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units [0.3277163122167433]
  We present PULSE-ICU, a self-supervised foundation model that learns event-level ICU representations from large-scale EHR sequences.<n>A unified embedding module encodes event identity, continuous values, units, and temporal attributes, while a Longformer-based encoder enables efficient modeling of long trajectories.
  arXiv  Detail & Related papers  (2025-11-27T08:10:52Z)
• Cross-Representation Benchmarking in Time-Series Electronic Health Records for Clinical Outcome Prediction [44.23284500920266]
  This benchmark standardises data curation and evaluation across two distinct clinical settings.<n>Experiments reveal that event stream models consistently deliver the strongest performance.<n>We find that feature selection strategies must be adapted to the clinical setting.
  arXiv  Detail & Related papers  (2025-10-10T09:03:47Z)
• Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech [51.14752758616364]
  Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments.<n>We propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework.<n>The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.
  arXiv  Detail & Related papers  (2025-10-05T09:32:12Z)
• CardiacCLIP: Video-based CLIP Adaptation for LVEF Prediction in a Few-shot Manner [14.429336783145644]
  Left ventricular ejection fraction (LVEF) serves as a key indicator of heart function.<n>Existing LVEF estimation methods depend on large-scale annotated video datasets.<n>We propose CardiacCLIP, a video-based framework that enhances LVEF prediction through attention-based frame aggregation and multi-resolution input scaling.
  arXiv  Detail & Related papers  (2025-09-21T12:52:08Z)
• Sensing Cardiac Health Across Scenarios and Devices: A Multi-Modal Foundation Model Pretrained on Heterogeneous Data from 1.7 Million Individuals [36.08910150609342]
  We present a cardiac sensing foundation model (CSFM) that learns unified representations from vast, heterogeneous health records.<n>Our model is pretrained on an innovative multi-modal integration of data from multiple large-scale datasets.<n> CSFM consistently outperforms traditional one-modal-one-task approaches.
  arXiv  Detail & Related papers  (2025-06-23T20:58:12Z)
• Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners [10.088785685439134]
  We propose D-BETA, a framework that pre-trains ECG and text data using a contrastive masked auto-encoder architecture.<n>D-BETA uniquely combines the strengths of generative with boosted discriminative capabilities to achieve robust cross-modal representations.
  arXiv  Detail & Related papers  (2024-10-03T01:24:09Z)
• MEDPSeg: Hierarchical polymorphic multitask learning for the segmentation of ground-glass opacities, consolidation, and pulmonary structures on computed tomography [37.119000111386924]
  MEDPSeg learns from heterogeneous chest CT targets through hierarchical polymorphic multitask learning (HPML) We show PML enabling new state-of-the-art performance for GGO and consolidation segmentation tasks. In addition, MEDPSeg simultaneously performs segmentation of the lung parenchyma, airways, pulmonary artery, and lung lesions, all in a single forward prediction.
  arXiv  Detail & Related papers  (2023-12-04T21:46:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.