Temporal Entailment Pretraining for Clinical Language Models over EHR Data

• URL: http://arxiv.org/abs/2504.18128v1
• Date: Fri, 25 Apr 2025 07:30:38 GMT
• Title: Temporal Entailment Pretraining for Clinical Language Models over EHR Data
• Authors: Tatsunori Tanaka, Fi Zheng, Kai Sato, Zhifeng Li, Yuanyun Zhang, Shi Li,
• Abstract summary: We introduce a novel temporal entailment pretraining objective for language models in the clinical domain.<n>Our method formulates EHR segments as temporally ordered sentence pairs and trains the model to determine whether a later state is entailed by, contradictory to, or neutral with respect to an earlier state.
• Score: 9.584923572354045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical language models have achieved strong performance on downstream tasks by pretraining on domain specific corpora such as discharge summaries and medical notes. However, most approaches treat the electronic health record as a static document, neglecting the temporally-evolving and causally entwined nature of patient trajectories. In this paper, we introduce a novel temporal entailment pretraining objective for language models in the clinical domain. Our method formulates EHR segments as temporally ordered sentence pairs and trains the model to determine whether a later state is entailed by, contradictory to, or neutral with respect to an earlier state. Through this temporally structured pretraining task, models learn to perform latent clinical reasoning over time, improving their ability to generalize across forecasting and diagnosis tasks. We pretrain on a large corpus derived from MIMIC IV and demonstrate state of the art results on temporal clinical QA, early warning prediction, and disease progression modeling.

Related papers

• Zero-shot Medical Event Prediction Using a Generative Pre-trained Transformer on Electronic Health Records [8.575985305475355]
  We show that a foundation model trained on EHRs can perform predictive tasks in a zero-shot manner.<n>Unlike supervised approaches requiring extensive labeled data, our method enables the model to forecast a next medical event purely from a pretraining knowledge.
  arXiv  Detail & Related papers  (2025-03-07T19:26:47Z)
• Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Prediction [10.403187385041702]
  We introduce MERA, a clinical diagnosis prediction model that bridges pertaining natural language knowledge with medical practice.<n>We apply hierarchical contrastive learning on a disease candidate ranking list to alleviate the large decision space issue.
  arXiv  Detail & Related papers  (2025-01-28T22:38:45Z)
• Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models [69.06149482021071]
  We propose a novel EHR data generation model called EHRPD. It is a diffusion-based model designed to predict the next visit based on the current one while also incorporating time interval estimation. We conduct experiments on two public datasets and evaluate EHRPD from fidelity, privacy, and utility perspectives.
  arXiv  Detail & Related papers  (2024-06-20T02:20:23Z)
• Intensive Care as One Big Sequence Modeling Problem [1.6114012813668932]
  We propose the paradigm of Healthcare as Sequence Modeling, in which interaction between the patient and the healthcare provider is represented as an event stream. We develop MIMIC-SEQ, a sequence modeling benchmark derived by translating heterogenous clinical records from MIMIC-IV dataset into a uniform event stream format.
  arXiv  Detail & Related papers  (2024-02-27T13:36:55Z)
• Recent Advances in Predictive Modeling with Electronic Health Records [71.19967863320647]
  utilizing EHR data for predictive modeling presents several challenges due to its unique characteristics. Deep learning has demonstrated its superiority in various applications, including healthcare.
  arXiv  Detail & Related papers  (2024-02-02T00:31:01Z)
• CPLLM: Clinical Prediction with Large Language Models [0.07083082555458872]
  We present a method that involves fine-tuning a pre-trained Large Language Model (LLM) for clinical disease and readmission prediction. For diagnosis prediction, we predict whether patients will be diagnosed with a target disease during their next visit or in the subsequent diagnosis, leveraging their historical diagnosis records. Our experiments have shown that our proposed method, CPLLM, surpasses all the tested models in terms of PR-AUC and ROC-AUC metrics.
  arXiv  Detail & Related papers  (2023-09-20T13:24:12Z)
• Textual Data Augmentation for Patient Outcomes Prediction [67.72545656557858]
  We propose a novel data augmentation method to generate artificial clinical notes in patients' Electronic Health Records. We fine-tune the generative language model GPT-2 to synthesize labeled text with the original training data. We evaluate our method on the most common patient outcome, i.e., the 30-day readmission rate.
  arXiv  Detail & Related papers  (2022-11-13T01:07:23Z)
• SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models [48.07469930813923]
  This work aims to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. We introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.
  arXiv  Detail & Related papers  (2021-08-31T08:23:56Z)
• Self-Supervised Graph Learning with Hyperbolic Embedding for Temporal Health Event Prediction [13.24834156675212]
  We propose a hyperbolic embedding method with information flow to pre-train medical code representations in a hierarchical structure. We incorporate these pre-trained representations into a graph neural network to detect disease complications. We present a new hierarchy-enhanced historical prediction proxy task in our self-supervised learning framework to fully utilize EHR data.
  arXiv  Detail & Related papers  (2021-06-09T00:42:44Z)
• Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration [55.88616573143478]
  Outcome prediction from clinical text can prevent doctors from overlooking possible risks. Diagnoses at discharge, procedures performed, in-hospital mortality and length-of-stay prediction are four common outcome prediction targets. We propose clinical outcome pre-training to integrate knowledge about patient outcomes from multiple public sources.
  arXiv  Detail & Related papers  (2021-02-08T10:26:44Z)
• MIA-Prognosis: A Deep Learning Framework to Predict Therapy Response [58.0291320452122]
  This paper aims at a unified deep learning approach to predict patient prognosis and therapy response. We formalize the prognosis modeling as a multi-modal asynchronous time series classification task. Our predictive model could further stratify low-risk and high-risk patients in terms of long-term survival.
  arXiv  Detail & Related papers  (2020-10-08T15:30:17Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.