BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data

• URL: http://arxiv.org/abs/2101.12037v1
• Date: Thu, 28 Jan 2021 14:54:01 GMT
• Title: BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data
• Authors: Demetres Kostas, Stephane Aroca-Ouellette, Frank Rudzicz
• Abstract summary: We consider how to adapt techniques and architectures used for language modelling (LM) to encephalography modelling (EM) We find that a single pre-trained model is capable of modelling completely novel raw EEG sequences recorded with differing hardware. Both the internal representations of this model and the entire architecture can be fine-tuned to a variety of downstream BCI and EEG classification tasks.
• Score: 15.71234837305808
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks (DNNs) used for brain-computer-interface (BCI) classification are commonly expected to learn general features when trained across a variety of contexts, such that these features could be fine-tuned to specific contexts. While some success is found in such an approach, we suggest that this interpretation is limited and an alternative would better leverage the newly (publicly) available massive EEG datasets. We consider how to adapt techniques and architectures used for language modelling (LM), that appear capable of ingesting awesome amounts of data, towards the development of encephalography modelling (EM) with DNNs in the same vein. We specifically adapt an approach effectively used for automatic speech recognition, which similarly (to LMs) uses a self-supervised training objective to learn compressed representations of raw data signals. After adaptation to EEG, we find that a single pre-trained model is capable of modelling completely novel raw EEG sequences recorded with differing hardware, and different subjects performing different tasks. Furthermore, both the internal representations of this model and the entire architecture can be fine-tuned to a variety of downstream BCI and EEG classification tasks, outperforming prior work in more task-specific (sleep stage classification) self-supervision.

Related papers

• Knowledge-guided EEG Representation Learning [27.8095014391814]
  Self-supervised learning has produced impressive results in multimedia domains of audio, vision and speech. We propose a self-supervised model for EEG, which provides robust performance and remarkable parameter efficiency. We also propose a novel knowledge-guided pre-training objective that accounts for the idiosyncrasies of the EEG signal.
  arXiv  Detail & Related papers  (2024-02-15T01:52:44Z)
• EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model [39.363511340878624]
  We present a novel EEG foundation model, namely EEGFormer, pretrained on large-scale compound EEG data. To validate the effectiveness of our model, we extensively evaluate it on various downstream tasks and assess the performance under different transfer settings.
  arXiv  Detail & Related papers  (2024-01-11T17:36:24Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• Analyzing EEG Data with Machine and Deep Learning: A Benchmark [23.893444154059324]
  This paper focuses on EEG signal analysis, and for the first time in literature, a benchmark of machine and deep learning for EEG signal classification. For our experiments we used the four most widespread models, i.e., multilayer perceptron, convolutional neural network, long short-term memory, and gated recurrent unit.
  arXiv  Detail & Related papers  (2022-03-18T15:18:55Z)
• BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search engines [49.75878234192369]
  We present WEAVER, a simple, yet efficient post-processing method that infuses old knowledge into the new model. We show that applying WEAVER in a sequential manner results in similar word embedding distributions as doing a combined training on all data at once.
  arXiv  Detail & Related papers  (2022-02-21T10:34:41Z)
• Recurrent Neural Networks with Mixed Hierarchical Structures and EM Algorithm for Natural Language Processing [9.645196221785694]
  We develop an approach called the latent indicator layer to identify and learn implicit hierarchical information. We also develop an EM algorithm to handle the latent indicator layer in training. We show that the EM-HRNN model with bootstrap training outperforms other RNN-based models in document classification tasks.
  arXiv  Detail & Related papers  (2022-01-21T23:08:33Z)
• Multi-Branch Deep Radial Basis Function Networks for Facial Emotion Recognition [80.35852245488043]
  We propose a CNN based architecture enhanced with multiple branches formed by radial basis function (RBF) units. RBF units capture local patterns shared by similar instances using an intermediate representation. We show it is the incorporation of local information what makes the proposed model competitive.
  arXiv  Detail & Related papers  (2021-09-07T21:05:56Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Uncovering the structure of clinical EEG signals with self-supervised learning [64.4754948595556]
  Supervised learning paradigms are often limited by the amount of labeled data that is available. This phenomenon is particularly problematic in clinically-relevant data, such as electroencephalography (EEG) By extracting information from unlabeled data, it might be possible to reach competitive performance with deep neural networks.
  arXiv  Detail & Related papers  (2020-07-31T14:34:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.