Label-Efficient Sleep Staging Using Transformers Pre-trained with Position Prediction

• URL: http://arxiv.org/abs/2404.15308v1
• Date: Fri, 29 Mar 2024 23:22:30 GMT
• Title: Label-Efficient Sleep Staging Using Transformers Pre-trained with Position Prediction
• Authors: Sayeri Lala, Hanlin Goh, Christopher Sandino,
• Abstract summary: We propose an architecture that seamlessly couples the feature and temporal encoding and a suitable pretraining scheme that pretrains the entire model. On a sample sleep staging dataset, we find that the proposed scheme offers performance gains that do not saturate with amount of labeled training data.
• Score: 2.591936982899312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sleep staging is a clinically important task for diagnosing various sleep disorders, but remains challenging to deploy at scale because it because it is both labor-intensive and time-consuming. Supervised deep learning-based approaches can automate sleep staging but at the expense of large labeled datasets, which can be unfeasible to procure for various settings, e.g., uncommon sleep disorders. While self-supervised learning (SSL) can mitigate this need, recent studies on SSL for sleep staging have shown performance gains saturate after training with labeled data from only tens of subjects, hence are unable to match peak performance attained with larger datasets. We hypothesize that the rapid saturation stems from applying a sub-optimal pretraining scheme that pretrains only a portion of the architecture, i.e., the feature encoder, but not the temporal encoder; therefore, we propose adopting an architecture that seamlessly couples the feature and temporal encoding and a suitable pretraining scheme that pretrains the entire model. On a sample sleep staging dataset, we find that the proposed scheme offers performance gains that do not saturate with amount of labeled training data (e.g., 3-5\% improvement in balanced sleep staging accuracy across low- to high-labeled data settings), reducing the amount of labeled training data needed for high performance (e.g., by 800 subjects). Based on our findings, we recommend adopting this SSL paradigm for subsequent work on SSL for sleep staging.

Related papers

• DailyMAE: Towards Pretraining Masked Autoencoders in One Day [37.206816999538496]
  Masked image modeling (MIM) has drawn attention for its effectiveness in learning data representation from unlabeled data. In this study, we propose efficient training recipes for MIM based SSL that focuses on mitigating data loading bottlenecks. Our library enables the training of a MAE-Base/16 model on the ImageNet 1K dataset for 800 epochs within just 18 hours.
  arXiv  Detail & Related papers  (2024-03-31T00:59:10Z)
• Quantifying the Impact of Data Characteristics on the Transferability of Sleep Stage Scoring Models [0.10878040851637998]
  Deep learning models for scoring sleep stages based on single-channel EEG have been proposed as a promising method for remote sleep monitoring. Applying these models to new datasets, particularly from wearable devices, raises two questions. First, when annotations on a target dataset are unavailable, which different data characteristics affect the sleep stage scoring performance the most and by how much? We propose a novel method for quantifying the impact of different data characteristics on the transferability of deep learning models.
  arXiv  Detail & Related papers  (2023-03-28T07:57:21Z)
• Self-supervised Learning for Label-Efficient Sleep Stage Classification: A Comprehensive Evaluation [13.895332825128076]
  Self-supervised learning (SSL) paradigm has shined as one of the most successful techniques to overcome the scarcity of labeled data. In this paper, we evaluate the efficacy of SSL to boost the performance of existing SSC models in the few-labels regime. We find that fine-tuning the pretrained SSC models with only 5% of labeled data can achieve competitive performance to the supervised training with full labels.
  arXiv  Detail & Related papers  (2022-10-10T09:01:17Z)
• Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audio [19.865050806327147]
  Self-supervised learning has proven vital in speech and audio-related applications. This paper provides the first empirical study of SSL pre-training for different specified sequence lengths. We find that training on short sequences can dramatically reduce resource costs while retaining a satisfactory performance for all tasks.
  arXiv  Detail & Related papers  (2022-09-30T16:35:42Z)
• Pseudo-Labeling Based Practical Semi-Supervised Meta-Training for Few-Shot Learning [93.63638405586354]
  We propose a simple and effective meta-training framework, called pseudo-labeling based meta-learning (PLML) Firstly, we train a classifier via common semi-supervised learning (SSL) and use it to obtain the pseudo-labels of unlabeled data. We build few-shot tasks from labeled and pseudo-labeled data and design a novel finetuning method with feature smoothing and noise suppression.
  arXiv  Detail & Related papers  (2022-07-14T10:53:53Z)
• Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routing [76.78772372631623]
  A common practice for self-supervised pre-training is to use as much data as possible. For a specific downstream task, however, involving irrelevant data in pre-training may degenerate the downstream performance. It is burdensome and infeasible to use different downstream-task-customized datasets in pre-training for different tasks.
  arXiv  Detail & Related papers  (2022-05-26T10:49:43Z)
• Open-Set Semi-Supervised Learning for 3D Point Cloud Understanding [62.17020485045456]
  It is commonly assumed in semi-supervised learning (SSL) that the unlabeled data are drawn from the same distribution as that of the labeled ones. We propose to selectively utilize unlabeled data through sample weighting, so that only conducive unlabeled data would be prioritized.
  arXiv  Detail & Related papers  (2022-05-02T16:09:17Z)
• DATA: Domain-Aware and Task-Aware Pre-training [94.62676913928831]
  We present DATA, a simple yet effective NAS approach specialized for self-supervised learning (SSL) Our method achieves promising results across a wide range of computation costs on downstream tasks, including image classification, object detection and semantic segmentation.
  arXiv  Detail & Related papers  (2022-03-17T02:38:49Z)
• Dash: Semi-Supervised Learning with Dynamic Thresholding [72.74339790209531]
  We propose a semi-supervised learning (SSL) approach that uses unlabeled examples to train models. Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection.
  arXiv  Detail & Related papers  (2021-09-01T23:52:29Z)
• Self-Tuning for Data-Efficient Deep Learning [75.34320911480008]
  Self-Tuning is a novel approach to enable data-efficient deep learning. It unifies the exploration of labeled and unlabeled data and the transfer of a pre-trained model. It outperforms its SSL and TL counterparts on five tasks by sharp margins.
  arXiv  Detail & Related papers  (2021-02-25T14:56:19Z)
• RobustSleepNet: Transfer learning for automated sleep staging at scale [0.0]
  Sleep disorder diagnosis relies on the analysis of polysomnography (PSG) records. In practice, sleep stage classification relies on the visual inspection of 30-seconds epochs of polysomnography signals. We introduce RobustSleepNet, a deep learning model for automatic sleep stage classification able to handle arbitrary PSG montages.
  arXiv  Detail & Related papers  (2021-01-07T09:39:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.