Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study

• URL: http://arxiv.org/abs/2605.12241v1
• Date: Tue, 12 May 2026 15:10:15 GMT
• Title: Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study
• Abstract summary: This work focuses on foundation models for electrocardiography (ECG) data, one of the most widely captured physiological time series world-wide.<n>We present a comprehensive assessment of pretraining methodologies, covering five different contrastive and non-contrastive self-supervised learning objectives for ECG foundation models.<n>We investigate their scaling behavior with pretraining dataset sizes up to 11M input samples, exclusively from publicly available sources.
• Score: 1.8964402635820148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Specialized foundation models are beginning to emerge in various medical subdomains, but pretraining methodologies and parametric scaling with the size of the pretraining dataset are rarely assessed systematically and in a like-for-like manner. This work focuses on foundation models for electrocardiography (ECG) data, one of the most widely captured physiological time series world-wide. We present a comprehensive assessment of pretraining methodologies, covering five different contrastive and non-contrastive self-supervised learning objectives for ECG foundation models, and investigate their scaling behavior with pretraining dataset sizes up to 11M input samples, exclusively from publicly available sources. Pretraining strategy has a meaningful and consistent impact on downstream performance, with contrastive predictive coding (slightly ahead of JEPA) yielding the most transferable representations across diverse clinical tasks. Scaling pretraining data continues to yield meaningful improvements up to 11M samples for most objectives. We also compare model architectures across all pretraining methodologies and find evidence for a clear superiority of structured state space models compared to transformers and CNN models. We hypothesize that the strong inductive biases of structured state space models, rather than pretraining scale alone, are the primary driver of effective ECG representation learning, with important implications for future foundation model development in this and potentially other physiological signal domains.

Related papers

• Towards Brain MRI Foundation Models for the Clinic: Findings from the FOMO25 Challenge [33.95846433611413]
  We organized the FOMO25 challenge as a satellite event at MICCAI 2025.<n>We provided participants with a large pretraining dataset, FOMO60K, and evaluated models on data sourced directly from clinical labels.<n>19 foundation models from sixteen teams were evaluated using a standardized containerized pipeline.
  arXiv  Detail & Related papers  (2026-04-13T16:23:46Z)
• Investigating the Impact of Histopathological Foundation Models on Regressive Prediction of Homologous Recombination Deficiency [52.50039435394964]
  We systematically evaluate foundation models for regression-based tasks.<n>We extract patch-level features from whole slide images (WSI) using five state-of-the-art foundation models.<n>Models are trained to predict continuous HRD scores based on these extracted features across breast, endometrial, and lung cancer cohorts.
  arXiv  Detail & Related papers  (2026-01-29T14:06:50Z)
• EEG Foundation Models: Progresses, Benchmarking, and Open Problems [10.447009984769819]
  We review 50 representative EEG foundation models and organize their design choices into a unified taxonomic framework.<n>We evaluate 12 open-source foundation models and competitive specialist baselines across 13 EEG datasets spanning nine BCI paradigms.
  arXiv  Detail & Related papers  (2026-01-25T15:28:50Z)
• Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals [22.364607686570384]
  Foundation models are large-scale machine learning models that are pre-trained on massive amounts of data.<n>Recent works, such as MOMENT, train a generalist time series foundation model with data from multiple domains.<n>This paper aims to conduct a comprehensive benchmarking study to compare the performance of generalist and specialist models.
  arXiv  Detail & Related papers  (2025-10-16T03:13:04Z)
• Exploring Scaling Laws for EHR Foundation Models [17.84205864956449]
  We present the first empirical investigation of scaling laws for EHR foundation models.<n>We identify consistent scaling patterns, including parabolic IsoFLOPs curves and power-law relationships between compute, model parameters, data size, and clinical utility.
  arXiv  Detail & Related papers  (2025-05-29T01:05:11Z)
• SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation [81.36747103102459]
  Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications.<n>Current state-of-the-art methods focus on training innovative architectural designs on confined datasets.<n>We investigate the impact of scaling up EHPS towards a family of generalist foundation models.
  arXiv  Detail & Related papers  (2025-01-16T18:59:46Z)
• The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis [27.310894780313618]
  This paper undertakes a comprehensive comparison of model capabilities at various pretraining intermediate checkpoints. We confirm that specific downstream metrics exhibit similar training dynamics across models of different sizes. In addition to our core findings, we've reproduced Amber and OpenLLaMA, releasing their intermediate checkpoints.
  arXiv  Detail & Related papers  (2024-04-01T16:00:01Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Scaling Representation Learning from Ubiquitous ECG with State-Space Models [28.776392386988043]
  We introduce textbfWildECG, a pre-trained state-space model for representation learning from ECG signals. We train this model in a self-supervised manner with 275,000 10s ECG recordings collected in the wild and evaluate it on a range of downstream tasks.
  arXiv  Detail & Related papers  (2023-09-26T22:08:19Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• Pre-training via Denoising for Molecular Property Prediction [53.409242538744444]
  We describe a pre-training technique that utilizes large datasets of 3D molecular structures at equilibrium. Inspired by recent advances in noise regularization, our pre-training objective is based on denoising.
  arXiv  Detail & Related papers  (2022-05-31T22:28:34Z)
• A multi-stage machine learning model on diagnosis of esophageal manometry [50.591267188664666]
  The framework includes deep-learning models at the swallow-level stage and feature-based machine learning models at the study-level stage. This is the first artificial-intelligence-style model to automatically predict CC diagnosis of HRM study from raw multi-swallow data.
  arXiv  Detail & Related papers  (2021-06-25T20:09:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.