Designing Pre-training Datasets from Unlabeled Data for EEG Classification with Transformers

• URL: http://arxiv.org/abs/2410.07190v1
• Date: Mon, 23 Sep 2024 13:26:13 GMT
• Title: Designing Pre-training Datasets from Unlabeled Data for EEG Classification with Transformers
• Authors: Tim Bary, Benoit Macq,
• Abstract summary: We present a way to design several labeled datasets from unlabeled electroencephalogram (EEG) data. These can then be used to pre-train transformers to learn representations of EEG signals. We tested this method on an epileptic seizure forecasting task on the Temple University Seizure Detection Corpus.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Transformer neural networks require a large amount of labeled data to train effectively. Such data is often scarce in electroencephalography, as annotations made by medical experts are costly. This is why self-supervised training, using unlabeled data, has to be performed beforehand. In this paper, we present a way to design several labeled datasets from unlabeled electroencephalogram (EEG) data. These can then be used to pre-train transformers to learn representations of EEG signals. We tested this method on an epileptic seizure forecasting task on the Temple University Seizure Detection Corpus using a Multi-channel Vision Transformer. Our results suggest that 1) Models pre-trained using our approach demonstrate significantly faster training times, reducing fine-tuning duration by more than 50% for the specific task, and 2) Pre-trained models exhibit improved accuracy, with an increase from 90.93% to 92.16%, as well as a higher AUC, rising from 0.9648 to 0.9702 when compared to non-pre-trained models.

Related papers

• 4,500 Seconds: Small Data Training Approaches for Deep UAV Audio Classification [2.3354223046061016]
  This study investigates deep learning approaches to UAV classification focusing on the key issue of data scarcity.<n>We train the models using a total of 4,500 seconds of audio samples, evenly distributed across a 9-class dataset.<n>We compare the use of convolutional neural networks (CNNs) and attention-based transformers.
  arXiv  Detail & Related papers  (2025-05-21T22:34:07Z)
• BioSerenity-E1: a self-supervised EEG model for medical applications [0.0]
  BioSerenity-E1 is a family of self-supervised foundation models for clinical EEG applications. It combines spectral tokenization with masked prediction to achieve state-of-the-art performance across relevant diagnostic tasks.
  arXiv  Detail & Related papers  (2025-03-13T13:42:46Z)
• Pre-Training Graph Contrastive Masked Autoencoders are Strong Distillers for EEG [4.006670302810497]
  We propose a Graph Contrastive Masked Autoencoder Distiller to bridge the gap between unlabeled/labeled and high/low-density EEG data. For knowledge distillation from high-density to low-density EEG data, we propose a Graph Topology Distillation loss function. We demonstrate the effectiveness of our method on four classification tasks across two clinical EEG datasets.
  arXiv  Detail & Related papers  (2024-11-28T15:53:32Z)
• Cross-Shaped Windows Transformer with Self-supervised Pretraining for Clinically Significant Prostate Cancer Detection in Bi-parametric MRI [6.930082824262643]
  We introduce a novel end-to-end Cross-Shaped windows (CSwin) transformer UNet model, CSwin UNet, to detect clinically significant prostate cancer (csPCa) in prostate bi-parametric MR imaging (bpMRI) Using a large prostate bpMRI dataset with 1500 patients, we first pretrain CSwin transformer using multi-task self-supervised learning to improve data-efficiency and network generalizability. Five-fold cross validation shows that self-supervised CSwin UNet achieves 0.888 AUC and 0.545 Average Precision (AP), significantly outperforming four comparable models (Swin U
  arXiv  Detail & Related papers  (2023-04-30T04:40:32Z)
• Convolutional Neural Networks for the classification of glitches in gravitational-wave data streams [52.77024349608834]
  We classify transient noise signals (i.e.glitches) and gravitational waves in data from the Advanced LIGO detectors. We use models with a supervised learning approach, both trained from scratch using the Gravity Spy dataset. We also explore a self-supervised approach, pre-training models with automatically generated pseudo-labels.
  arXiv  Detail & Related papers  (2023-03-24T11:12:37Z)
• Efficiently Training Vision Transformers on Structural MRI Scans for Alzheimer's Disease Detection [2.359557447960552]
  Vision transformers (ViT) have emerged in recent years as an alternative to CNNs for several computer vision applications. We tested variants of the ViT architecture for a range of desired neuroimaging downstream tasks based on difficulty. We achieved a performance boost of 5% and 9-10% upon fine-tuning vision transformer models pre-trained on synthetic and real MRI scans.
  arXiv  Detail & Related papers  (2023-03-14T20:18:12Z)
• Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training [110.79400526706081]
  Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference. This paper proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT.
  arXiv  Detail & Related papers  (2022-11-19T21:15:47Z)
• Improving self-supervised pretraining models for epileptic seizure detection from EEG data [0.23624125155742057]
  This paper presents various self-supervision strategies to enhance the performance of a time-series based Diffusion convolution neural network (DCRNN) model. The learned weights in the self-supervision pretraining phase can be transferred to the supervised training phase to boost the model's prediction capability.
  arXiv  Detail & Related papers  (2022-06-28T17:15:49Z)
• Towards physiology-informed data augmentation for EEG-based BCIs [24.15108821320151]
  We suggest a novel technique for augmenting the training data by generating new data from the data set at hand. In this manuscript, we explain the method and show first preliminary results for participant-independent motor-imagery classification.
  arXiv  Detail & Related papers  (2022-03-27T20:59:40Z)
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification [54.55281692768765]
  Transformer-based supervised pre-training achieves great performance in person re-identification (ReID) Due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset to boost the performance. This work aims to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure.
  arXiv  Detail & Related papers  (2021-11-23T18:59:08Z)
• SOUL: An Energy-Efficient Unsupervised Online Learning Seizure Detection Classifier [68.8204255655161]
  Implantable devices that record neural activity and detect seizures have been adopted to issue warnings or trigger neurostimulation to suppress seizures. For an implantable seizure detection system, a low power, at-the-edge, online learning algorithm can be employed to dynamically adapt to neural signal drifts. SOUL was fabricated in TSMC's 28 nm process occupying 0.1 mm2 and achieves 1.5 nJ/classification energy efficiency, which is at least 24x more efficient than state-of-the-art.
  arXiv  Detail & Related papers  (2021-10-01T23:01:20Z)
• How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers? [7.727662147015879]
  Task-agnostic forms of data augmentation have proven widely effective in computer vision, even on pretrained models. We ask how effective these techniques really are when applied to pretrained transformers. We observe a negative result, finding that techniques which previously reported strong improvements for non-pretrained models fail to consistently improve performance for pretrained transformers.
  arXiv  Detail & Related papers  (2020-10-05T03:55:15Z)
• Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
  We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training. We experimentally verify that the new dataset can significantly improve the ability of the learned FER model. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
  arXiv  Detail & Related papers  (2020-05-18T09:36:51Z)
• Radioactive data: tracing through training [130.2266320167683]
  We propose a new technique, emphradioactive data, that makes imperceptible changes to this dataset such that any model trained on it will bear an identifiable mark. Given a trained model, our technique detects the use of radioactive data and provides a level of confidence (p-value) Our method is robust to data augmentation and backdoority of deep network optimization.
  arXiv  Detail & Related papers  (2020-02-03T18:41:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.