Temporal Entailment Pretraining for Clinical Language Models over EHR Data

• URL: http://arxiv.org/abs/2504.18128v1
• Date: Fri, 25 Apr 2025 07:30:38 GMT
• Title: Temporal Entailment Pretraining for Clinical Language Models over EHR Data
• Abstract summary: We introduce a novel temporal entailment pretraining objective for language models in the clinical domain.<n>Our method formulates EHR segments as temporally ordered sentence pairs and trains the model to determine whether a later state is entailed by, contradictory to, or neutral with respect to an earlier state.
• Score: 9.584923572354045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical language models have achieved strong performance on downstream tasks by pretraining on domain specific corpora such as discharge summaries and medical notes. However, most approaches treat the electronic health record as a static document, neglecting the temporally-evolving and causally entwined nature of patient trajectories. In this paper, we introduce a novel temporal entailment pretraining objective for language models in the clinical domain. Our method formulates EHR segments as temporally ordered sentence pairs and trains the model to determine whether a later state is entailed by, contradictory to, or neutral with respect to an earlier state. Through this temporally structured pretraining task, models learn to perform latent clinical reasoning over time, improving their ability to generalize across forecasting and diagnosis tasks. We pretrain on a large corpus derived from MIMIC IV and demonstrate state of the art results on temporal clinical QA, early warning prediction, and disease progression modeling.

Related papers

• Reinforcement Learning over Patient Trajectories for Clinical Reasoning in EHR Foundation Models [38.37812950780674]
  We propose a reinforcement learning framework that treats EHR foundation models as generative policies over patient trajectories.<n>We formulate common clinical prediction problems as event-conditioned, time-windowed reasoning tasks.<n>We then design time-aware, rollout-sensitive rewards to account for finite rollout lengths and temporally inconclusive outcomes.
  arXiv  Detail & Related papers  (2026-09-10T23:10:38Z)
• ReMedi: Reasoner for Medical Clinical Prediction [70.84466325266068]
  We present ReMedi, a framework for improving clinical outcome prediction from EHR.<n>ReMedi generates rationale-answer pairs using a challenging sample regeneration mechanism for complex clinical questions.<n> Experiments on multiple EHR prediction tasks demonstrate substantial gains of up to 19.9 percent over state-of-the-art baselines in terms of F1 score.
  arXiv  Detail & Related papers  (2026-05-02T14:44:49Z)
• Clinically Interpretable Sepsis Early Warning via LLM-Guided Simulation of Temporal Physiological Dynamics [24.96692273948327]
  Timely and interpretable early warning of sepsis remains a clinical challenge.<n>Traditional data-driven models provide accurate yet opaque predictions.<n>We propose a Large Language Model (LLM)-guided temporal simulation framework.
  arXiv  Detail & Related papers  (2026-04-22T06:30:54Z)
• The Taxonomies, Training, and Applications of Event Stream Modelling for Electronic Health Records [6.63303513347406]
  EHRs offer transformative potential for artificial intelligence in healthcare.<n>Traditional modelling approaches often struggle to accommodate the inherent sparsity and irregularity of real-world clinical data.<n>Research has shifted toward event stream representation, which treats patient records as continuous sequences.
  arXiv  Detail & Related papers  (2026-03-14T16:08:30Z)
• CLARITY: Medical World Model for Guiding Treatment Decisions by Modeling Context-Aware Disease Trajectories in Latent Space [49.74032713886216]
  CLARITY is a medical world model that forecasts disease evolution directly within a structured latent space.<n>It explicitly integrates time intervals (temporal context) and patient-specific data (clinical context) to model treatment-conditioned progression as a smooth, interpretable trajectory.
  arXiv  Detail & Related papers  (2025-12-08T20:42:10Z)
• Language Models for Longitudinal Clinical Prediction [2.5925656171325127]
  We explore a lightweight framework that adapts frozen large language models to analyze longitudinal clinical data.<n>The approach integrates patient history and context within the language model space to generate accurate forecasts without model fine-tuning.<n>It achieves accurate and reliable performance even with minimal training data, showing promise for early-stage Alzheimer's monitoring.
  arXiv  Detail & Related papers  (2025-10-27T21:49:03Z)
• Serialized EHR make for good text representations [1.585843510099207]
  SerialBEHRT is a domain aligned foundation model that extends SciBERT through additional pretraining on structured EHR sequences.<n>We evaluate its effectiveness on the task of antibiotic susceptibility prediction, a clinically meaningful problem in antibiotic stewardship.
  arXiv  Detail & Related papers  (2025-10-11T17:16:15Z)
• Building the EHR Foundation Model via Next Event Prediction [5.378917071184147]
  Next Event Prediction (NEP) is a framework that enhances Large Language Models' temporal reasoning.<n>NEP explicitly models disease progression patterns and causal relationships.<n>Our analyses reveal dual benefits: state-of-the-art prediction accuracy combined with clinically interpretable attention patterns.
  arXiv  Detail & Related papers  (2025-09-29T23:27:51Z)
• Predictive Multimodal Modeling of Diagnoses and Treatments in EHR [10.29848511141116]
  We propose a system to fuse clinical notes and events captured in electronic health records.<n>The model integrates pre-trained encoders, feature pooling, and cross-modal attention to learn optimal representations across modalities.<n> Experiments show that these strategies enhance the early prediction model, outperforming the current state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-08-14T22:14:18Z)
• Time-Aware Attention for Enhanced Electronic Health Records Modeling [8.4225455796455]
  TALE-EHR is a Transformer-based framework featuring a novel time-aware attention mechanism that explicitly models continuous temporal gaps.<n>Our approach outperforms state-of-the-art baselines on tasks such as disease progression forecasting.
  arXiv  Detail & Related papers  (2025-07-20T07:32:41Z)
• Zero-shot Medical Event Prediction Using a Generative Pre-trained Transformer on Electronic Health Records [8.575985305475355]
  We show that a foundation model trained on EHRs can perform predictive tasks in a zero-shot manner.<n>Unlike supervised approaches requiring extensive labeled data, our method enables the model to forecast a next medical event purely from a pretraining knowledge.
  arXiv  Detail & Related papers  (2025-03-07T19:26:47Z)
• Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Prediction [10.403187385041702]
  We introduce MERA, a clinical diagnosis prediction model that bridges pertaining natural language knowledge with medical practice.<n>We apply hierarchical contrastive learning on a disease candidate ranking list to alleviate the large decision space issue.
  arXiv  Detail & Related papers  (2025-01-28T22:38:45Z)
• CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis [50.56875995511431]
  We introduce a Cross-Modal Temporal Pattern Discovery (CTPD) framework, designed to efficiently extract meaningful cross-modal temporal patterns from multimodal EHR data.<n>Our approach introduces shared initial temporal pattern representations which are refined using slot attention to generate temporal semantic embeddings.
  arXiv  Detail & Related papers  (2024-11-01T15:54:07Z)
• Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models [69.06149482021071]
  We propose a novel EHR data generation model called EHRPD. It is a diffusion-based model designed to predict the next visit based on the current one while also incorporating time interval estimation. We conduct experiments on two public datasets and evaluate EHRPD from fidelity, privacy, and utility perspectives.
  arXiv  Detail & Related papers  (2024-06-20T02:20:23Z)
• Intensive Care as One Big Sequence Modeling Problem [1.6114012813668932]
  We propose the paradigm of Healthcare as Sequence Modeling, in which interaction between the patient and the healthcare provider is represented as an event stream. We develop MIMIC-SEQ, a sequence modeling benchmark derived by translating heterogenous clinical records from MIMIC-IV dataset into a uniform event stream format.
  arXiv  Detail & Related papers  (2024-02-27T13:36:55Z)
• Recent Advances in Predictive Modeling with Electronic Health Records [71.19967863320647]
  utilizing EHR data for predictive modeling presents several challenges due to its unique characteristics. Deep learning has demonstrated its superiority in various applications, including healthcare.
  arXiv  Detail & Related papers  (2024-02-02T00:31:01Z)
• CPLLM: Clinical Prediction with Large Language Models [0.07083082555458872]
  We present a method that involves fine-tuning a pre-trained Large Language Model (LLM) for clinical disease and readmission prediction. For diagnosis prediction, we predict whether patients will be diagnosed with a target disease during their next visit or in the subsequent diagnosis, leveraging their historical diagnosis records. Our experiments have shown that our proposed method, CPLLM, surpasses all the tested models in terms of PR-AUC and ROC-AUC metrics.
  arXiv  Detail & Related papers  (2023-09-20T13:24:12Z)
• Textual Data Augmentation for Patient Outcomes Prediction [67.72545656557858]
  We propose a novel data augmentation method to generate artificial clinical notes in patients' Electronic Health Records. We fine-tune the generative language model GPT-2 to synthesize labeled text with the original training data. We evaluate our method on the most common patient outcome, i.e., the 30-day readmission rate.
  arXiv  Detail & Related papers  (2022-11-13T01:07:23Z)
• SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models [48.07469930813923]
  This work aims to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. We introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.
  arXiv  Detail & Related papers  (2021-08-31T08:23:56Z)
• Self-Supervised Graph Learning with Hyperbolic Embedding for Temporal Health Event Prediction [13.24834156675212]
  We propose a hyperbolic embedding method with information flow to pre-train medical code representations in a hierarchical structure. We incorporate these pre-trained representations into a graph neural network to detect disease complications. We present a new hierarchy-enhanced historical prediction proxy task in our self-supervised learning framework to fully utilize EHR data.
  arXiv  Detail & Related papers  (2021-06-09T00:42:44Z)
• Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration [55.88616573143478]
  Outcome prediction from clinical text can prevent doctors from overlooking possible risks. Diagnoses at discharge, procedures performed, in-hospital mortality and length-of-stay prediction are four common outcome prediction targets. We propose clinical outcome pre-training to integrate knowledge about patient outcomes from multiple public sources.
  arXiv  Detail & Related papers  (2021-02-08T10:26:44Z)
• MIA-Prognosis: A Deep Learning Framework to Predict Therapy Response [58.0291320452122]
  This paper aims at a unified deep learning approach to predict patient prognosis and therapy response. We formalize the prognosis modeling as a multi-modal asynchronous time series classification task. Our predictive model could further stratify low-risk and high-risk patients in terms of long-term survival.
  arXiv  Detail & Related papers  (2020-10-08T15:30:17Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.