Is Self-Pretraining really useful to improve diagnosis in medical Time Series?

• URL: http://arxiv.org/abs/2608.06122v2
• Date: Fri, 07 Aug 2026 08:30:22 GMT
• Title: Is Self-Pretraining really useful to improve diagnosis in medical Time Series?
• Abstract summary: Self-PreTraining (SPT) improves classification accuracy by 0-6 percentage points depending on masking strategy, dataset and architecture.<n>SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes.
• Score: 15.692598361763055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series. Our objective is to assess the impact of SPT on the performance and scalability of transformer-based models across diverse medical applications, particularly under limited data conditions. We evaluate transformer architectures on three representative medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease). Models are trained either from scratch or through SPT using four masking-based objectives designed to promote temporal and cross-modal representation learning, and we systematically vary model depth to examine how capacity interacts with pre-training benefits. Across datasets and configurations, SPT consistently improves classification accuracy by 0-6 percentage points depending on masking strategy, dataset and architecture, with gains observed not only in multivariate settings but also when models are restricted to simple univariate inputs. The improvements increase for deeper models that can better exploit the enriched temporal representations learned during pre-training. These findings indicate that SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes, supporting its potential to improve robustness and accuracy in data-limited clinical settings.

Related papers

• Benchmarking Machine Learning Architectures for Antimicrobial Stewardship in Pediatric ICUs [1.113021697417606]
  Antimicrobial stewardship (AMS) is critical in pediatric intensive care units (PICUs)<n>Machine learning offers a promising approach for identifying patient-level opportunities for stewardship interventions from electronic health record data.<n>We present a systematic benchmarking study of AMS intervention prediction in the PICU across a public dataset and a private institutional cohort.
  arXiv  Detail & Related papers  (2026-05-21T15:26:58Z)
• Adaptive Test-Time Training for Predicting Need for Invasive Mechanical Ventilation in Multi-Center Cohorts [4.250825424649631]
  Test-Time Training (TTT) has emerged as a promising approach to mitigate such shifts.<n>We introduce Adaptive Test-Time Training (AdaTTT), an enhanced TTT framework tailored for EHR-based IMV prediction in ICU settings.
  arXiv  Detail & Related papers  (2025-12-07T04:27:40Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• METHOD: Modular Efficient Transformer for Health Outcome Discovery [0.25112747242081457]
  This paper introduces METHOD, a novel transformer architecture specifically designed to address the challenges of clinical sequence modelling in electronic health records.<n>METHODintegrates three key innovations: (1) a patient-aware attention mechanism that prevents information leakage whilst enabling efficient batch processing; (2) an adaptive sliding window attention scheme that captures multi-scale temporal dependencies; and (3) a U-Net inspired architecture with dynamic skip connections for effective long sequence processing.<n> Evaluations on the MIMIC-IV database demonstrate that METHODconsistently outperforms the state-of-the-art ETHOSmodel
  arXiv  Detail & Related papers  (2025-05-16T15:52:56Z)
• Self-Supervised Pre-training Tasks for an fMRI Time-series Transformer in Autism Detection [3.665816629105171]
  Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that encompasses a wide variety of symptoms and degrees of impairment. We have developed a transformer-based self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity. We show that randomly masking entire ROIs gives better model performance than randomly masking time points in the pre-training step.
  arXiv  Detail & Related papers  (2024-09-18T20:29:23Z)
• PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation [51.509573838103854]
  We propose a semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation. Our PMT generates high-fidelity pseudo labels by learning robust and diverse features in the training process. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches.
  arXiv  Detail & Related papers  (2024-09-08T15:02:25Z)
• MPRE: Multi-perspective Patient Representation Extractor for Disease Prediction [3.914545513460964]
  We propose the Multi-perspective Patient Representation Extractor (MPRE) for disease prediction. Specifically, we propose Frequency Transformation Module (FTM) to extract the trend and variation information of dynamic features. In the 2D Multi-Extraction Network (2D MEN), we form the 2D temporal tensor based on trend and variation. We also propose the First-Order Difference Attention Mechanism (FODAM) to calculate the contributions of differences in adjacent variations to the disease diagnosis.
  arXiv  Detail & Related papers  (2024-01-01T13:52:05Z)
• TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series [57.4208255711412]
  Building on copula theory, we propose a simplified objective for the recently-introduced transformer-based attentional copulas (TACTiS) We show that the resulting model has significantly better training dynamics and achieves state-of-the-art performance across diverse real-world forecasting tasks.
  arXiv  Detail & Related papers  (2023-10-02T16:45:19Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Robust and Efficient Medical Imaging with Self-Supervision [80.62711706785834]
  We present REMEDIS, a unified representation learning strategy to improve robustness and data-efficiency of medical imaging AI. We study a diverse range of medical imaging tasks and simulate three realistic application scenarios using retrospective data.
  arXiv  Detail & Related papers  (2022-05-19T17:34:18Z)
• SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models [48.07469930813923]
  This work aims to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. We introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.
  arXiv  Detail & Related papers  (2021-08-31T08:23:56Z)
• Self-Training with Improved Regularization for Sample-Efficient Chest X-Ray Classification [80.00316465793702]
  We present a deep learning framework that enables robust modeling in challenging scenarios. Our results show that using 85% lesser labeled data, we can build predictive models that match the performance of classifiers trained in a large-scale data setting.
  arXiv  Detail & Related papers  (2020-05-03T02:36:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.