MEIT: Multi-Modal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation

• URL: http://arxiv.org/abs/2403.04945v3
• Date: Tue, 18 Jun 2024 07:15:09 GMT
• Title: MEIT: Multi-Modal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation
• Authors: Zhongwei Wan, Che Liu, Xin Wang, Chaofan Tao, Hui Shen, Zhenwu Peng, Jie Fu, Rossella Arcucci, Huaxiu Yao, Mi Zhang,
• Abstract summary: Electrocardiogram (ECG) is the primary non-invasive diagnostic tool for monitoring cardiac conditions. Recent studies have concentrated on classifying cardiac conditions using ECG data but have overlooked ECG report generation. We propose the Multimodal ECG Instruction Tuning (MEIT) framework, the first attempt to tackle ECG report generation with LLMs and multimodal instructions.
• Score: 41.324530807795256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Electrocardiogram (ECG) is the primary non-invasive diagnostic tool for monitoring cardiac conditions and is crucial in assisting clinicians. Recent studies have concentrated on classifying cardiac conditions using ECG data but have overlooked ECG report generation, which is time-consuming and requires clinical expertise. To automate ECG report generation and ensure its versatility, we propose the Multimodal ECG Instruction Tuning (MEIT) framework, the first attempt to tackle ECG report generation with LLMs and multimodal instructions. To facilitate future research, we establish a benchmark to evaluate MEIT with various LLMs backbones across two large-scale ECG datasets. Our approach uniquely aligns the representations of the ECG signal and the report, and we conduct extensive experiments to benchmark MEIT with nine open-source LLMs using more than 800,000 ECG reports. MEIT's results underscore the superior performance of instruction-tuned LLMs, showcasing their proficiency in quality report generation, zero-shot capabilities, and resilience to signal perturbation. These findings emphasize the efficacy of our MEIT framework and its potential for real-world clinical application.

Related papers

• Self-supervised inter-intra period-aware ECG representation learning for detecting atrial fibrillation [41.82319894067087]
  We propose an inter-intra period-aware ECG representation learning approach. Considering ECGs of atrial fibrillation patients exhibit the irregularity in RR intervals and the absence of P-waves, we develop specific pre-training tasks for interperiod and intraperiod representations. Our approach demonstrates remarkable AUC performances on the BTCH dataset, textiti.e., 0.953/0.996 for paroxysmal/persistent atrial fibrillation detection.
  arXiv  Detail & Related papers  (2024-10-08T10:03:52Z)
• An Electrocardiogram Foundation Model Built on over 10 Million Recordings with External Evaluation across Multiple Domains [17.809094003643523]
  We introduce an ECG Foundation Model (ECGFounder) to broaden the diagnostic capabilities of ECG analysis. ECGFounder was trained on over 10 million ECGs with 150 label categories from the Harvard-Emory ECG Database. It achieves expert-level performance on internal validation sets, with AUROC exceeding 0.95 for eighty diagnoses.
  arXiv  Detail & Related papers  (2024-10-05T12:12:02Z)
• Electrocardiogram Report Generation and Question Answering via Retrieval-Augmented Self-Supervised Modeling [19.513904491604794]
  ECG-ReGen is a retrieval-based approach for ECG-to-text report generation and question answering. By combining pre-training with dynamic retrieval and Large Language Model (LLM)-based refinement, ECG-ReGen effectively analyzes ECG data and answers related queries.
  arXiv  Detail & Related papers  (2024-09-13T12:50:36Z)
• ECG-FM: An Open Electrocardiogram Foundation Model [3.611746032873298]
  We present ECG-FM, an open foundation model for ECG analysis. ECG-FM adopts a transformer-based architecture and is pretrained on 2.5 million samples. We show how its command of contextual information results in strong performance, rich pretrained embeddings, and reliable interpretability.
  arXiv  Detail & Related papers  (2024-08-09T17:06:49Z)
• Multi-scale Cross-restoration Framework for Electrocardiogram Anomaly Detection [33.48389041651675]
  Electrocardiogram (ECG) is a widely used diagnostic tool for detecting heart conditions. Rare cardiac diseases may be underdiagnosed using traditional ECG analysis, considering that no training dataset can exhaust all possible cardiac disorders. This paper proposes using anomaly detection to identify any unhealthy status, with normal ECGs solely for training.
  arXiv  Detail & Related papers  (2023-08-03T09:16:57Z)
• PulseNet: Deep Learning ECG-signal classification using random augmentation policy and continous wavelet transform for canines [46.09869227806991]
  evaluating canine electrocardiograms (ECG) require skilled veterinarians. Current availability of veterinary cardiologists for ECG interpretation and diagnostic support is limited. We implement a deep convolutional neural network (CNN) approach for classifying canine electrocardiogram sequences as either normal or abnormal.
  arXiv  Detail & Related papers  (2023-05-17T09:06:39Z)
• Frozen Language Model Helps ECG Zero-Shot Learning [12.974685769614062]
  We propose Multimodal ECG-Text Self-supervised pre-training (METS) We use a trainable ECG encoder and a frozen language model to embed paired ECG and automatically machine-generated clinical reports separately. In downstream classification tasks, METS achieves around 10% improvement in performance without using any annotated data.
  arXiv  Detail & Related papers  (2023-03-22T05:01:14Z)
• SEVGGNet-LSTM: a fused deep learning model for ECG classification [38.747030782394646]
  The input ECG signals are firstly segmented and normalized, and then fed into the combined VGG and LSTM network for feature extraction and classification. An attention mechanism (SE block) is embedded into the core network for increasing the weight of important features.
  arXiv  Detail & Related papers  (2022-10-31T07:36:48Z)
• Leveraging Statistical Shape Priors in GAN-based ECG Synthesis [3.3482093430607267]
  We propose a novel approach for ECG signal generation using Generative Adversarial Networks (GANs) and statistical ECG data modeling. Our approach leverages prior knowledge about ECG dynamics to synthesize realistic signals, addressing the complex dynamics of ECG signals. Our results demonstrate that our approach, which models temporal and amplitude variations of ECG signals as 2-D shapes, generates more realistic signals compared to state-of-the-art GAN based generation baselines.
  arXiv  Detail & Related papers  (2022-10-22T18:06:11Z)
• ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed Quality Labeling Using Neural Networks [69.25956542388653]
  Deep learning (DL) algorithms are gaining weight in academic and industrial settings. We demonstrate DL can be successfully applied to low interpretative tasks by embedding ECG detection and delineation onto a segmentation framework. The model was trained using PhysioNet's QT database, comprised of 105 ambulatory ECG recordings.
  arXiv  Detail & Related papers  (2020-05-11T16:29:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.