How Do Electrocardiogram Models Scale?

• URL: http://arxiv.org/abs/2605.17276v1
• Date: Sun, 17 May 2026 05:53:35 GMT
• Title: How Do Electrocardiogram Models Scale?
• Abstract summary: We systematically investigate neural and loss-to-loss scaling laws within the electrocardiogram domain.<n>We decouple the effects of model architecture (ResNet vs. Transformer) and pre-training paradigm, namely supervised learning (SL) versus self-supervised learning (SSL)<n>Our results suggest that the path to effective ECG foundation models lies in the strategic alignment of architecture and paradigm rather than brute-force scaling.
• Score: 19.332418587904375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While scaling laws have established a fundamental framework for foundation models in natural language processing, their applicability to electrocardiogram (ECG) models remains poorly characterized. Indeed, recent studies do not always yield consistent downstream gains as one increases the model size or pre-training dataset size of ECG models, leaving the exact roles of architectural inductive biases, pre-training paradigms, and expected improvements with size largely unanswered. In this work, we systematically investigate neural and loss-to-loss scaling laws within the ECG domain. By pre-training over $120$ models (ranging from $20$K to $200$M parameters) on the large-scale CODE dataset ($2.3$M records), we decouple the effects of model architecture (ResNet vs. Transformer) and pre-training paradigm, namely supervised learning (SL) versus self-supervised learning (SSL). We found that (i) SL models are data-bottlenecked in-distribution, whereas SSL models scale robustly across both model and data sizes; (ii) for out-of-distribution (OOD) generalization, ResNets are $1.3$ to $2.5$ times more parameter-efficient than Transformers, while SSL is up to $16$ times more data-efficient and achieves up to $7.6$ times higher transfer efficiency than SL on unseen clinical tasks; (iii) across the observed scales, ResNet-based models generally achieve the lowest OOD loss, with SSL dominating on unseen clinical tasks and self-supervised Transformers overtaking at very large model sizes. Our results suggest that the path to effective ECG foundation models lies in the strategic alignment of architecture and paradigm rather than brute-force scaling.

Related papers

• Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning [19.12792297140574]
  Continual learning aims to incrementally train a model on a sequence of tasks while retaining performance on prior ones.<n> storing and replaying data is often infeasible due to privacy or security constraints.<n>We propose Per-layer Model Inversion (PMI), inspired by faster convergence in single-layer optimization.
  arXiv  Detail & Related papers  (2025-10-30T09:58:48Z)
• Towards a Comprehensive Scaling Law of Mixture-of-Experts [54.117786590884776]
  We propose a comprehensive and precise joint MoE scaling law that considers all essential factors.<n>Our results demonstrate that the optimal settings for $G$ and $S$ are independent of both the model architecture and data size.<n>Our proposed MoE scaling law could function as an accurate and insightful guidance to facilitate future MoE model design and training.
  arXiv  Detail & Related papers  (2025-09-28T06:35:34Z)
• Exploring Scaling Laws for EHR Foundation Models [17.84205864956449]
  We present the first empirical investigation of scaling laws for EHR foundation models.<n>We identify consistent scaling patterns, including parabolic IsoFLOPs curves and power-law relationships between compute, model parameters, data size, and clinical utility.
  arXiv  Detail & Related papers  (2025-05-29T01:05:11Z)
• SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation [81.36747103102459]
  Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications.<n>Current state-of-the-art methods focus on training innovative architectural designs on confined datasets.<n>We investigate the impact of scaling up EHPS towards a family of generalist foundation models.
  arXiv  Detail & Related papers  (2025-01-16T18:59:46Z)
• Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget [53.882492597728195]
  This paper investigates how to efficiently train speech foundation models with self-supervised learning (SSL) under a limited compute budget.<n>We examine critical factors in SSL that impact the budget, including model architecture, model size, and data size.
  arXiv  Detail & Related papers  (2024-09-09T10:36:42Z)
• PanGu-$\pi$: Enhancing Language Model Architectures via Nonlinearity Compensation [97.78045712375047]
  We present a new efficient model architecture for large language models (LLMs) We show that PanGu-$pi$-7B can achieve a comparable performance to that of benchmarks with about 10% inference speed-up. In addition, we have deployed PanGu-$pi$-7B in the high-value domains of finance and law, developing an LLM named YunShan for practical application.
  arXiv  Detail & Related papers  (2023-12-27T11:49:24Z)
• LEMON: Lossless model expansion [43.40389747029802]
  Scaling of deep neural networks, especially Transformers, is pivotal for their surging performance. We present $textbfL$ossl$textbfE$ss $textbfMO$del Expansio$textbfN$ (LEMON), a recipe to initialize scaled models using the weights of their smaller but pre-trained counterparts. We show that LEMON reduces computational costs by 56.7% for Vision Transformers and 33.2% for BERT when compared to training from scratch.
  arXiv  Detail & Related papers  (2023-10-12T03:02:41Z)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning [52.29522018586365]
  We study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains.
  arXiv  Detail & Related papers  (2023-10-10T15:13:30Z)
• METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals [151.3601429216877]
  We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. We propose a recipe, namely "Model generated dEnoising TRaining Objective" (METRO) The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks.
  arXiv  Detail & Related papers  (2022-04-13T21:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.