EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model

• URL: http://arxiv.org/abs/2401.10278v1
• Date: Thu, 11 Jan 2024 17:36:24 GMT
• Title: EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model
• Authors: Yuqi Chen, Kan Ren, Kaitao Song, Yansen Wang, Yifan Wang, Dongsheng Li, Lili Qiu
• Abstract summary: We present a novel EEG foundation model, namely EEGFormer, pretrained on large-scale compound EEG data. To validate the effectiveness of our model, we extensively evaluate it on various downstream tasks and assess the performance under different transfer settings.
• Score: 39.363511340878624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised learning has emerged as a highly effective approach in the fields of natural language processing and computer vision. It is also applicable to brain signals such as electroencephalography (EEG) data, given the abundance of available unlabeled data that exist in a wide spectrum of real-world medical applications ranging from seizure detection to wave analysis. The existing works leveraging self-supervised learning on EEG modeling mainly focus on pretraining upon each individual dataset corresponding to a single downstream task, which cannot leverage the power of abundant data, and they may derive sub-optimal solutions with a lack of generalization. Moreover, these methods rely on end-to-end model learning which is not easy for humans to understand. In this paper, we present a novel EEG foundation model, namely EEGFormer, pretrained on large-scale compound EEG data. The pretrained model cannot only learn universal representations on EEG signals with adaptable performance on various downstream tasks but also provide interpretable outcomes of the useful patterns within the data. To validate the effectiveness of our model, we extensively evaluate it on various downstream tasks and assess the performance under different transfer settings. Furthermore, we demonstrate how the learned model exhibits transferable anomaly detection performance and provides valuable interpretability of the acquired patterns via self-supervised learning.

Related papers

• Dataset Refinement for Improving the Generalization Ability of the EEG Decoding Model [2.9972387721489655]
  We propose a dataset refinement algorithm to eliminate noisy data from EEG datasets. The proposed algorithm consistently led to better generalization performance compared to using the original dataset. We conclude that removing noisy data from the training dataset alone can effectively improve the generalization performance of deep learning models in the EEG domain.
  arXiv  Detail & Related papers  (2024-10-31T05:08:24Z)
• EEGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training [9.57946371147345]
  EEGPT is the first generalist EEG foundation model designed to address these challenges. First, we propose an electrode-wise modeling strategy that treats each electrode as a fundamental unit. Second, we develop the first autoregressive EEG pre-trained model. Third, we introduce a multi-task transfer learning paradigm using a learnable electrode graph network.
  arXiv  Detail & Related papers  (2024-10-14T12:17:54Z)
• Neuro-GPT: Towards A Foundation Model for EEG [0.04188114563181615]
  We propose Neuro-GPT, a foundation model consisting of an EEG encoder and a GPT model. Foundation model is pre-trained on a large-scale data set using a self-supervised task that learns how to reconstruct masked EEG segments. Experiments demonstrate that applying a foundation model can significantly improve classification performance compared to a model trained from scratch.
  arXiv  Detail & Related papers  (2023-11-07T07:07:18Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• Cluster-level pseudo-labelling for source-free cross-domain facial expression recognition [94.56304526014875]
  We propose the first Source-Free Unsupervised Domain Adaptation (SFUDA) method for Facial Expression Recognition (FER) Our method exploits self-supervised pretraining to learn good feature representations from the target data. We validate the effectiveness of our method in four adaptation setups, proving that it consistently outperforms existing SFUDA methods when applied to FER.
  arXiv  Detail & Related papers  (2022-10-11T08:24:50Z)
• Exploiting Multiple EEG Data Domains with Adversarial Learning [20.878816519635304]
  We propose an adversarial inference approach to learn data-source invariant representations in this context. We unify EEG recordings from different source domains (i.e., emotion recognition SEED, SEED-IV, DEAP, DREAMER)
  arXiv  Detail & Related papers  (2022-04-16T11:09:20Z)
• BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search engines [49.75878234192369]
  We present WEAVER, a simple, yet efficient post-processing method that infuses old knowledge into the new model. We show that applying WEAVER in a sequential manner results in similar word embedding distributions as doing a combined training on all data at once.
  arXiv  Detail & Related papers  (2022-02-21T10:34:41Z)
• BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data [15.71234837305808]
  We consider how to adapt techniques and architectures used for language modelling (LM) to encephalography modelling (EM) We find that a single pre-trained model is capable of modelling completely novel raw EEG sequences recorded with differing hardware. Both the internal representations of this model and the entire architecture can be fine-tuned to a variety of downstream BCI and EEG classification tasks.
  arXiv  Detail & Related papers  (2021-01-28T14:54:01Z)
• Uncovering the structure of clinical EEG signals with self-supervised learning [64.4754948595556]
  Supervised learning paradigms are often limited by the amount of labeled data that is available. This phenomenon is particularly problematic in clinically-relevant data, such as electroencephalography (EEG) By extracting information from unlabeled data, it might be possible to reach competitive performance with deep neural networks.
  arXiv  Detail & Related papers  (2020-07-31T14:34:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.