ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

• URL: http://arxiv.org/abs/2603.14326v1
• Date: Sun, 15 Mar 2026 11:09:38 GMT
• Title: ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation
• Abstract summary: We introduce a novel multi-turn evaluation framework comprising over 6,400 samples to assess step-by-step reasoning across 17 core ECG diagnoses.<n>Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction.<n>These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI.
• Score: 14.296134981531337
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce \textbf{ECG-Reasoning-Benchmark}, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.

Related papers

• ECG-LENS: Lead-Aware Clinical Context Enriched ECG Report Generation and Evaluation [5.4061821164321175]
  ECG report generation could reduce clinicians' interpretive workload, improve diagnostic efficiency, and expand access to cardiac assessment in underserved communities.<n>We propose ECG-LENS, an end-to-end ECG report-generation framework that integrates multi-lead signal modeling, diagnosis-aware representations, and clinically grounded text generation.
  arXiv  Detail & Related papers  (2026-08-06T11:18:31Z)
• ECG-LLM: Foundation Model for ECG-Based Cardiac Reasoning [50.82313761980655]
  ECG-LLM is an ECG-conditioned large language model trained across four cohorts 679,112 ECG studies from 186,409 patients.<n>It is trained on clinically structured question-answer pairs derived from ECG signals, clinical context, CMR, and ECHO.<n>It successfully recovers conventional ECG measurements, such as heart rate, and strongly predicts complex CMR-derived phenotypes.
  arXiv  Detail & Related papers  (2026-07-15T11:06:35Z)
• Cross-Modal Contrastive Learning of ECG and Angiography Representations for Severe Stenosis Classification [79.18126160626093]
  Coronary artery stenosis is a common cardiovascular disease, with severe, untreated cases posing significant risks of heart attack.<n>We introduce StenCE, a pretraining framework, allowing stratification of patients based on features derived directly from ECGs.<n>The obtained models successfully detect signals for stenosis diagnosis in ECGs and are the first to achieve high performance in severe stenosis classification.
  arXiv  Detail & Related papers  (2026-05-23T14:50:22Z)
• Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs [63.535652574541764]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis.<n>Their application in gastrointestinal endoscopy is currently hindered by two critical limitations.<n>We propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-21T07:47:37Z)
• Position: Evaluation of ECG Representations Must Be Fixed [1.567009619451362]
  This position paper argues that current benchmarking practice in 12-lead ECG representation learning must be fixed to ensure progress is reliable and aligned with clinically meaningful objectives.<n>We argue that downstream evaluation should expand to include an assessment of structural heart disease and patient-level forecasting, in addition to other evolving ECG-related endpoints, as relevant clinical targets.
  arXiv  Detail & Related papers  (2026-02-19T16:42:46Z)
• UniPACT: A Multimodal Framework for Prognostic Question Answering on Raw ECG and Structured EHR [15.533178197005208]
  We propose a unified framework for prognostic question answering that bridges this modality gap.<n>UniPACT's core contribution is a structured prompting mechanism that converts numerical EHR data into semantically rich text.<n>We evaluate it on the comprehensive MDS-ED benchmark, it achieves a state-of-the-art mean AUROC of 89.37% across a diverse set of prognostic tasks.
  arXiv  Detail & Related papers  (2026-01-25T17:35:52Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• Interpretable Multimodal Zero-Shot ECG Diagnosis via Structured Clinical Knowledge Alignment [10.34278624026891]
  ZETA is a zero-shot multimodal framework designed for interpretable ECG diagnosis aligned with clinical knowledge.<n>We will release the curated observation dataset and code to facilitate future research.
  arXiv  Detail & Related papers  (2025-10-24T15:09:09Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• CardioRAG: A Retrieval-Augmented Generation Framework for Multimodal Chagas Disease Detection [3.2889108396912974]
  Chagas disease affects nearly 6 million people worldwide, with Chagas cardiomyopathy representing its most severe complication.<n>In regions where serological testing capacity is limited, AI-enhanced electrocardiogram (ECG) screening provides a critical diagnostic alternative.<n>We propose a retrieval-augmented generation framework, CardioRAG, integrating large language models with interpretable ECG-based clinical features.
  arXiv  Detail & Related papers  (2025-10-02T01:02:04Z)
• Signal, Image, or Symbolic: Exploring the Best Input Representation for Electrocardiogram-Language Models Through a Unified Framework [18.95201514457046]
  Large language models (LLMs) have been applied to electrocardiogram (ECG) interpretation.<n>Electrocardiogram-Language Models (ELMs) emulate expert cardiac electrophysiologists by issuing diagnoses, analyzing waveform morphology, identifying contributing factors, and proposing patient-specific action plans.<n>We present the first comprehensive benchmark of these modalities across 6 public datasets and 5 evaluation metrics.
  arXiv  Detail & Related papers  (2025-05-24T19:43:15Z)
• GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images [44.50428701650495]
  We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation.<n> GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations.<n>We propose the Grounded ECG task, a clinically motivated benchmark designed to assess the MLLM's capability in grounded ECG understanding.
  arXiv  Detail & Related papers  (2025-03-08T05:48:53Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)
• ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed Quality Labeling Using Neural Networks [69.25956542388653]
  Deep learning (DL) algorithms are gaining weight in academic and industrial settings. We demonstrate DL can be successfully applied to low interpretative tasks by embedding ECG detection and delineation onto a segmentation framework. The model was trained using PhysioNet's QT database, comprised of 105 ambulatory ECG recordings.
  arXiv  Detail & Related papers  (2020-05-11T16:29:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.