CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings

• URL: http://arxiv.org/abs/2501.18891v1
• Date: Fri, 31 Jan 2025 05:00:02 GMT
• Title: CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings
• Authors: Mohammad Al Olaimat, Serdar Bozdag,
• Abstract summary: We introduce CAAT-EHR, a novel architecture designed to generate task-agnostic longitudinal embeddings from raw EHR data. An autoregressive decoder complements the encoder by predicting future time points data during pre-training, ensuring that the resulting embeddings maintain temporal consistency and alignment.
• Score: 0.0
• License:
• Abstract: Electronic health records (EHRs) provide a comprehensive source of longitudinal patient data, encompassing structured modalities such as laboratory results, imaging data, and vital signs, and unstructured clinical notes. These datasets, after necessary preprocessing to clean and format the data for analysis, often remain in their raw EHR form, representing numerical or categorical values without further transformation into task-agnostic embeddings. While such raw EHR data enables predictive modeling, its reliance on manual feature engineering or downstream task-specific optimization limits its utility for general-purpose applications. Deep learning (DL) techniques, such as recurrent neural networks (RNNs) and Transformers, have facilitated predictive tasks like disease progression and diagnosis prediction. However, these methods often struggle to fully exploit the temporal and multimodal dependencies inherent in EHR data due to their reliance on pre-processed but untransformed raw EHR inputs. In this study, we introduce CAAT-EHR, a novel architecture designed to bridge this gap by generating robust, task-agnostic longitudinal embeddings from raw EHR data. CAAT-EHR leverages self- and cross-attention mechanisms in its encoder to integrate temporal and contextual relationships across multiple modalities, transforming the data into enriched embeddings that capture complex dependencies. An autoregressive decoder complements the encoder by predicting future time points data during pre-training, ensuring that the resulting embeddings maintain temporal consistency and alignment. CAAT-EHR eliminates the need for manual feature engineering and enables seamless transferability across diverse downstream tasks. Extensive evaluations on benchmark datasets, demonstrate the superiority of CAAT-EHR-generated embeddings over pre-processed raw EHR data and other baseline approaches.

Related papers

• Self-Supervised Pre-Training with Joint-Embedding Predictive Architecture Boosts ECG Classification Performance [0.0]
  We create a large unsupervised pre-training dataset by combining ten public ECG databases. We pre-train Vision Transformers using JEPA on this dataset and fine-tune them on various PTB-XL benchmarks.
  arXiv  Detail & Related papers  (2024-10-02T08:25:57Z)
• Dynamic Data Pruning for Automatic Speech Recognition [58.95758272440217]
  We introduce Dynamic Data Pruning for ASR (DDP-ASR), which offers fine-grained pruning granularities specifically tailored for speech-related datasets. Our experiments show that DDP-ASR can save up to 1.6x training time with negligible performance loss.
  arXiv  Detail & Related papers  (2024-06-26T14:17:36Z)
• Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models [69.06149482021071]
  We propose a novel EHR data generation model called EHRPD. It is a diffusion-based model designed to predict the next visit based on the current one while also incorporating time interval estimation. We conduct experiments on two public datasets and evaluate EHRPD from fidelity, privacy, and utility perspectives.
  arXiv  Detail & Related papers  (2024-06-20T02:20:23Z)
• Computationally and Memory-Efficient Robust Predictive Analytics Using Big Data [0.0]
  This study navigates through the challenges of data uncertainties, storage limitations, and predictive data-driven modeling using big data. We utilize Robust Principal Component Analysis (RPCA) for effective noise reduction and outlier elimination, and Optimal Sensor Placement (OSP) for efficient data compression and storage.
  arXiv  Detail & Related papers  (2024-03-27T22:39:08Z)
• Knowledge Enhanced Conditional Imputation for Healthcare Time-series [9.937117045677923]
  Conditional Self-Attention Imputation (CSAI) is a novel recurrent neural network architecture designed to address the challenges of complex missing data patterns. CSAI extends the current state-of-the-art neural network-based imputation methods by introducing key modifications specifically adapted to EHR data characteristics. This work significantly advances the state of neural network imputation applied to EHRs by more closely aligning algorithmic imputation with clinical realities.
  arXiv  Detail & Related papers  (2023-12-27T20:42:40Z)
• MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation [58.93221876843639]
  This paper introduces a novel, end-to-end diffusion-based risk prediction model, named MedDiffusion. It enhances risk prediction performance by creating synthetic patient data during training to enlarge sample space. It discerns hidden relationships between patient visits using a step-wise attention mechanism, enabling the model to automatically retain the most vital information for generating high-quality data.
  arXiv  Detail & Related papers  (2023-10-04T01:36:30Z)
• Unsupervised Pre-Training on Patient Population Graphs for Patient-Level Predictions [48.02011627390706]
  Pre-training has shown success in different areas of machine learning, such as Computer Vision (CV), Natural Language Processing (NLP) and medical imaging. In this paper, we apply unsupervised pre-training to heterogeneous, multi-modal EHR data for patient outcome prediction. We find that our proposed graph based pre-training method helps in modeling the data at a population level.
  arXiv  Detail & Related papers  (2022-03-23T17:59:45Z)
• Generating Synthetic Mixed-type Longitudinal Electronic Health Records for Artificial Intelligent Applications [9.374416143268892]
  generative adversarial network (GAN) entitled EHR-M-GAN which synthesizes textitmixed-type timeseries EHR data. We have validated EHR-M-GAN on three publicly-available intensive care unit databases with records from a total of 141,488 unique patients.
  arXiv  Detail & Related papers  (2021-12-22T17:17:34Z)
• SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models [48.07469930813923]
  This work aims to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. We introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.
  arXiv  Detail & Related papers  (2021-08-31T08:23:56Z)
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data [62.29031007761901]
  This paper proposes a novel contrastive regularized clinical classification model. We introduce two unique positive sampling strategies specifically tailored for EHR data. Our framework yields highly competitive experimental results in predicting the mortality risk on real-world COVID-19 EHR data.
  arXiv  Detail & Related papers  (2021-04-07T06:02:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.