De-biased Multimodal Electrocardiogram Analysis

• URL: http://arxiv.org/abs/2411.14795v1
• Date: Fri, 22 Nov 2024 08:35:35 GMT
• Title: De-biased Multimodal Electrocardiogram Analysis
• Authors: Haitao Li, Ziyu Li, Yiheng Mao, Ziyi Liu, Zhoujian Sun, Zhengxing Huang,
• Abstract summary: Multimodal large language models (MLLMs) are increasingly being applied in the medical field. Previous studies have attempted to address this by converting ECGs into several text tags. In this work, we directly feed the embeddings of ECGs into the LLM through a projection layer.
• Score: 20.290531515033518
• License:
• Abstract: Multimodal large language models (MLLMs) are increasingly being applied in the medical field, particularly in medical imaging. However, developing MLLMs for ECG signals, which are crucial in clinical settings, has been a significant challenge beyond medical imaging. Previous studies have attempted to address this by converting ECGs into several text tags using an external classifier in a training-free manner. However, this approach significantly compresses the information in ECGs and underutilizes the reasoning capabilities of LLMs. In this work, we directly feed the embeddings of ECGs into the LLM through a projection layer, retaining more information about ECGs and better leveraging the reasoning abilities of LLMs. Our method can also effectively handle a common situation in clinical practice where it is necessary to compare two ECGs taken at different times. Recent studies found that MLLMs may rely solely on text input to provide answers, ignoring inputs from other modalities. We analyzed this phenomenon from a causal perspective in the context of ECG MLLMs and discovered that the confounder, severity of illness, introduces a spurious correlation between the question and answer, leading the model to rely on this spurious correlation and ignore the ECG input. Such models do not comprehend the ECG input and perform poorly in adversarial tests where different expressions of the same question are used in the training and testing sets. We designed a de-biased pre-training method to eliminate the confounder's effect according to the theory of backdoor adjustment. Our model performed well on the ECG-QA task under adversarial testing and demonstrated zero-shot capabilities. An interesting random ECG test further validated that our model effectively understands and utilizes the input ECG signal.

Related papers

• Teach Multimodal LLMs to Comprehend Electrocardiographic Images [10.577263066644194]
  We introduce ECGInstruct, a comprehensive ECG image instruction tuning dataset of over one million samples. We also develop PULSE, an MLLM tailored for ECG image comprehension. Our experiments show that PULSE sets a new state-of-the-art, outperforming general MLLMs with an average accuracy improvement of 15% to 30%.
  arXiv  Detail & Related papers  (2024-10-21T20:26:41Z)
• Electrocardiogram-Language Model for Few-Shot Question Answering with Meta Learning [19.513904491604794]
  Electrocardiogram (ECG) interpretation requires specialized expertise. This work introduces a novel multimodal meta-learning method for few-shot ECG question answering.
  arXiv  Detail & Related papers  (2024-10-18T13:48:01Z)
• Self-supervised inter-intra period-aware ECG representation learning for detecting atrial fibrillation [41.82319894067087]
  We propose an inter-intra period-aware ECG representation learning approach. Considering ECGs of atrial fibrillation patients exhibit the irregularity in RR intervals and the absence of P-waves, we develop specific pre-training tasks for interperiod and intraperiod representations. Our approach demonstrates remarkable AUC performances on the BTCH dataset, textiti.e., 0.953/0.996 for paroxysmal/persistent atrial fibrillation detection.
  arXiv  Detail & Related papers  (2024-10-08T10:03:52Z)
• Electrocardiogram Report Generation and Question Answering via Retrieval-Augmented Self-Supervised Modeling [19.513904491604794]
  ECG-ReGen is a retrieval-based approach for ECG-to-text report generation and question answering. By combining pre-training with dynamic retrieval and Large Language Model (LLM)-based refinement, ECG-ReGen effectively analyzes ECG data and answers related queries.
  arXiv  Detail & Related papers  (2024-09-13T12:50:36Z)
• ECG-FM: An Open Electrocardiogram Foundation Model [3.611746032873298]
  We present ECG-FM, an open foundation model for ECG analysis. ECG-FM adopts a transformer-based architecture and is pretrained on 2.5 million samples. We show how its command of contextual information results in strong performance, rich pretrained embeddings, and reliable interpretability.
  arXiv  Detail & Related papers  (2024-08-09T17:06:49Z)
• MEIT: Multi-Modal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation [41.324530807795256]
  Electrocardiogram (ECG) is the primary non-invasive diagnostic tool for monitoring cardiac conditions. Recent studies have concentrated on classifying cardiac conditions using ECG data but have overlooked ECG report generation. We propose the Multimodal ECG Instruction Tuning (MEIT) framework, the first attempt to tackle ECG report generation with LLMs and multimodal instructions.
  arXiv  Detail & Related papers  (2024-03-07T23:20:56Z)
• PulseNet: Deep Learning ECG-signal classification using random augmentation policy and continous wavelet transform for canines [46.09869227806991]
  evaluating canine electrocardiograms (ECG) require skilled veterinarians. Current availability of veterinary cardiologists for ECG interpretation and diagnostic support is limited. We implement a deep convolutional neural network (CNN) approach for classifying canine electrocardiogram sequences as either normal or abnormal.
  arXiv  Detail & Related papers  (2023-05-17T09:06:39Z)
• SEVGGNet-LSTM: a fused deep learning model for ECG classification [38.747030782394646]
  The input ECG signals are firstly segmented and normalized, and then fed into the combined VGG and LSTM network for feature extraction and classification. An attention mechanism (SE block) is embedded into the core network for increasing the weight of important features.
  arXiv  Detail & Related papers  (2022-10-31T07:36:48Z)
• Continuous Decoding of Daily-Life Hand Movements from Forearm Muscle Activity for Enhanced Myoelectric Control of Hand Prostheses [78.120734120667]
  We introduce a novel method, based on a long short-term memory (LSTM) network, to continuously map forearm EMG activity onto hand kinematics. Ours is the first reported work on the prediction of hand kinematics that uses this challenging dataset. Our results suggest that the presented method is suitable for the generation of control signals for the independent and proportional actuation of the multiple DOFs of state-of-the-art hand prostheses.
  arXiv  Detail & Related papers  (2021-04-29T00:11:32Z)
• ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed Quality Labeling Using Neural Networks [69.25956542388653]
  Deep learning (DL) algorithms are gaining weight in academic and industrial settings. We demonstrate DL can be successfully applied to low interpretative tasks by embedding ECG detection and delineation onto a segmentation framework. The model was trained using PhysioNet's QT database, comprised of 105 ambulatory ECG recordings.
  arXiv  Detail & Related papers  (2020-05-11T16:29:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.