Looking Beyond Accuracy: A Holistic Benchmark of ECG Foundation Models

• URL: http://arxiv.org/abs/2601.21830v1
• Date: Thu, 29 Jan 2026 15:14:00 GMT
• Title: Looking Beyond Accuracy: A Holistic Benchmark of ECG Foundation Models
• Authors: Francesca Filice, Edoardo De Rose, Simone Bartucci, Francesco Calimeri, Simona Perri,
• Abstract summary: This study aims to find an in-depth, comprehensive benchmarking framework for Foundation Models (FMs)<n>We introduce a benchmark methodology that complements performance-based evaluation with representation-level analysis.<n>We also rely on the methodology for carrying out an extensive evaluation of several ECG-expert FMs pretrained via state-of-the-art techniques.
• Score: 0.3914676152740142
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The electrocardiogram (ECG) is a cost-effective, highly accessible and widely employed diagnostic tool. With the advent of Foundation Models (FMs), the field of AI-assisted ECG interpretation has begun to evolve, as they enable model reuse across different tasks by relying on embeddings. However, to responsibly employ FMs, it is crucial to rigorously assess to which extent the embeddings they produce are generalizable, particularly in error-sensitive domains such as healthcare. Although prior works have already addressed the problem of benchmarking ECG-expert FMs, they focus predominantly on the evaluation of downstream performance. To fill this gap, this study aims to find an in-depth, comprehensive benchmarking framework for FMs, with a specific focus on ECG-expert ones. To this aim, we introduce a benchmark methodology that complements performance-based evaluation with representation-level analysis, leveraging SHAP and UMAP techniques. Furthermore, we rely on the methodology for carrying out an extensive evaluation of several ECG-expert FMs pretrained via state-of-the-art techniques over different cross-continental datasets and data availability settings; this includes ones featuring data scarcity, a fairly common situation in real-world medical scenarios. Experimental results show that our benchmarking protocol provides a rich insight of ECG-expert FMs' embedded patterns, enabling a deeper understanding of their representational structure and generalizability.

Related papers

• Position: Evaluation of ECG Representations Must Be Fixed [1.567009619451362]
  This position paper argues that current benchmarking practice in 12-lead ECG representation learning must be fixed to ensure progress is reliable and aligned with clinically meaningful objectives.<n>We argue that downstream evaluation should expand to include an assessment of structural heart disease and patient-level forecasting, in addition to other evolving ECG-related endpoints, as relevant clinical targets.
  arXiv  Detail & Related papers  (2026-02-19T16:42:46Z)
• Transferring Clinical Knowledge into ECGs Representation [0.19498378931702776]
  We propose a novel three-stage training paradigm that transfers knowledge from multimodal clinical data into a powerful, yet unimodal, ECG encoder.<n>We employ a self-supervised, joint-embedding pre-training stage to create an ECG representation that is enriched with contextual clinical information.<n>As an indirect way to explain the model's output, we train it to also predict associated laboratory abnormalities directly from the ECG embedding.
  arXiv  Detail & Related papers  (2025-12-07T22:19:24Z)
• EnECG: Efficient Ensemble Learning for Electrocardiogram Multi-task Foundation Model [46.84040404474695]
  EnECG is an ensemble-based framework that integrates multiple specialized foundation models, each excelling in different aspects of ECG interpretation.<n>We show that EnECG can help reduce computational and memory costs while maintaining the strong representational power of foundation models.<n>This framework not only enhances feature extraction and predictive performance but also ensures practical efficiency for real-world clinical applications.
  arXiv  Detail & Related papers  (2025-11-28T07:22:33Z)
• An Electrocardiogram Multi-task Benchmark with Comprehensive Evaluations and Insightful Findings [21.836042030973797]
  Analyzing the ECG typically requires domain expertise, which is a roadblock to applying artificial intelligence for healthcare.<n>We evaluate language/general time-series/ECG foundation models in comparison with time-series deep learning models.<n>In-depth analyses and insights are provided along with comprehensive experimental results.
  arXiv  Detail & Related papers  (2025-11-28T06:47:21Z)
• Simulator and Experience Enhanced Diffusion Model for Comprehensive ECG Generation [52.19347532840774]
  We propose SE-Diff, a novel physiological simulator and experience enhanced diffusion model for ECG generation.<n> SE-Diff integrates a lightweight ordinary differential equation (ODE)-based ECG simulator into the diffusion process via a beat decoder.<n>Extensive experiments on real-world ECG datasets demonstrate that SE-Diff improves both signal fidelity and text-ECG semantic alignment.
  arXiv  Detail & Related papers  (2025-11-13T02:57:10Z)
• RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis [56.373297358647655]
  Retrieval-Augmented Diagnosis (RAD) is a novel framework that injects external knowledge into multimodal models directly on downstream tasks.<n>RAD operates through three key mechanisms: retrieval and refinement of disease-centered knowledge from multiple medical sources, a guideline-enhanced contrastive loss transformer, and a dual decoder.
  arXiv  Detail & Related papers  (2025-09-24T10:36:14Z)
• EEG-MedRAG: Enhancing EEG-based Clinical Decision-Making via Hierarchical Hypergraph Retrieval-Augmented Generation [45.031633614714]
  EEG-MedRAG is a three-layer hypergraph-based retrieval-augmented generation framework.<n>It unifies EEG domain knowledge, individual patient cases, and a large-scale repository into a traversable n-ary relational hypergraph.<n>We introduce the first cross-disease, cross-role EEG clinical QA benchmark, spanning seven disorders and five authentic clinical perspectives.
  arXiv  Detail & Related papers  (2025-08-19T11:12:58Z)
• Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models [52.2001050216955]
  Existing methods aim to enhance the performance of Medical Vision Language Model (MedVLM) by adjusting model structure, fine-tuning with high-quality data, or through preference fine-tuning.<n>We propose an expert-in-the-loop framework named Expert-Controlled-Free Guidance (Expert-CFG) to align MedVLM with clinical expertise without additional training.
  arXiv  Detail & Related papers  (2025-07-12T09:03:30Z)
• GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images [44.50428701650495]
  We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation.<n> GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations.<n>We propose the Grounded ECG task, a clinically motivated benchmark designed to assess the MLLM's capability in grounded ECG understanding.
  arXiv  Detail & Related papers  (2025-03-08T05:48:53Z)
• Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering [70.44269982045415]
  Retrieval-augmented generation (RAG) has emerged as a promising approach to enhance the performance of large language models (LLMs) We introduce Medical Retrieval-Augmented Generation Benchmark (MedRGB) that provides various supplementary elements to four medical QA datasets. Our experimental results reveals current models' limited ability to handle noise and misinformation in the retrieved documents.
  arXiv  Detail & Related papers  (2024-11-14T06:19:18Z)
• ECG-FM: An Open Electrocardiogram Foundation Model [3.8270632390229777]
  We present ECG-FM, an open foundation model for ECG analysis, and conduct a study using a dataset of 1.5 million ECGs.<n>ECG-FM is a transformer-based model pretrained using a hybrid contrastive and generative self-supervised learning approach.<n>We affirm that ECG-FM is robust, label-efficient, and functionally discriminative by showcasing data scaling experiments, performing a latent space analysis, and generating saliency maps.
  arXiv  Detail & Related papers  (2024-08-09T17:06:49Z)
• Unsupervised Pre-Training Using Masked Autoencoders for ECG Analysis [4.3312979375047025]
  This paper proposes an unsupervised pre-training technique based on masked autoencoder (MAE) for electrocardiogram (ECG) signals. In addition, we propose a task-specific fine-tuning to form a complete framework for ECG analysis. The framework is high-level, universal, and not individually adapted to specific model architectures or tasks.
  arXiv  Detail & Related papers  (2023-10-17T11:19:51Z)
• ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed Quality Labeling Using Neural Networks [69.25956542388653]
  Deep learning (DL) algorithms are gaining weight in academic and industrial settings. We demonstrate DL can be successfully applied to low interpretative tasks by embedding ECG detection and delineation onto a segmentation framework. The model was trained using PhysioNet's QT database, comprised of 105 ambulatory ECG recordings.
  arXiv  Detail & Related papers  (2020-05-11T16:29:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.