NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals

• URL: http://arxiv.org/abs/2409.00101v1
• Date: Tue, 27 Aug 2024 12:07:09 GMT
• Title: NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals
• Authors: Wei-Bang Jiang, Yansen Wang, Bao-Liang Lu, Dongsheng Li,
• Abstract summary: We propose NeuroLM, the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals as a foreign language. Our approach begins with learning a text-aligned neural tokenizer through vector-quantized temporal-frequency prediction, which encodes EEG signals into discrete neural tokens. We are the first to demonstrate that, by specific incorporation with LLMs, NeuroLM unifies diverse EEG tasks within a single model through instruction tuning.
• Score: 21.363722751437066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements for large-scale pre-training with neural signals such as electroencephalogram (EEG) have shown promising results, significantly boosting the development of brain-computer interfaces (BCIs) and healthcare. However, these pre-trained models often require full fine-tuning on each downstream task to achieve substantial improvements, limiting their versatility and usability, and leading to considerable resource wastage. To tackle these challenges, we propose NeuroLM, the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals as a foreign language, endowing the model with multi-task learning and inference capabilities. Our approach begins with learning a text-aligned neural tokenizer through vector-quantized temporal-frequency prediction, which encodes EEG signals into discrete neural tokens. These EEG tokens, generated by the frozen vector-quantized (VQ) encoder, are then fed into an LLM that learns causal EEG information via multi-channel autoregression. Consequently, NeuroLM can understand both EEG and language modalities. Finally, multi-task instruction tuning adapts NeuroLM to various downstream tasks. We are the first to demonstrate that, by specific incorporation with LLMs, NeuroLM unifies diverse EEG tasks within a single model through instruction tuning. The largest variant NeuroLM-XL has record-breaking 1.7B parameters for EEG signal processing, and is pre-trained on a large-scale corpus comprising approximately 25,000-hour EEG data. When evaluated on six diverse downstream datasets, NeuroLM showcases the huge potential of this multi-task learning paradigm.

Related papers

• Interpretable Language Modeling via Induction-head Ngram Models [74.26720927767398]
  We propose Induction-head ngram models (Induction-Gram) to bolster modern ngram models with a hand-engineered "induction head" This induction head uses a custom neural similarity metric to efficiently search the model's input context for potential next-word completions. Experiments show that this simple method significantly improves next-word prediction over baseline interpretable models.
  arXiv  Detail & Related papers  (2024-10-31T12:33:26Z)
• EEGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training [9.57946371147345]
  EEGPT is the first generalist EEG foundation model designed to address these challenges. First, we propose an electrode-wise modeling strategy that treats each electrode as a fundamental unit. Second, we develop the first autoregressive EEG pre-trained model. Third, we introduce a multi-task transfer learning paradigm using a learnable electrode graph network.
  arXiv  Detail & Related papers  (2024-10-14T12:17:54Z)
• CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models [68.64605538559312]
  In this paper, we analyze the MLLM instruction tuning from both theoretical and empirical perspectives. Inspired by our findings, we propose a measurement to quantitatively evaluate the learning balance. In addition, we introduce an auxiliary loss regularization method to promote updating of the generation distribution of MLLMs.
  arXiv  Detail & Related papers  (2024-07-29T23:18:55Z)
• Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models [24.54139799413152]
  We introduce a groundbreaking end-to-end (E2E) framework for decoding invasive brain signals. Our findings underscore the immense potential of E2E frameworks in speech neuroprosthesis.
  arXiv  Detail & Related papers  (2024-06-17T14:04:18Z)
• Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI [6.926908480247951]
  We propose a unified foundation model for EEG called Large Brain Model (LaBraM) LaBraM enables cross-dataset learning by segmenting the EEG signals into EEG channel patches. We then pre-train neural Transformers by predicting the original neural codes for the masked EEG channel patches.
  arXiv  Detail & Related papers  (2024-05-29T05:08:16Z)
• In-Context Language Learning: Architectures and Algorithms [73.93205821154605]
  We study ICL through the lens of a new family of model problems we term in context language learning (ICLL) We evaluate a diverse set of neural sequence models on regular ICLL tasks.
  arXiv  Detail & Related papers  (2024-01-23T18:59:21Z)
• Predicting Infant Brain Connectivity with Federated Multi-Trajectory GNNs using Scarce Data [54.55126643084341]
  Existing deep learning solutions suffer from three major limitations. We introduce FedGmTE-Net++, a federated graph-based multi-trajectory evolution network. Using the power of federation, we aggregate local learnings among diverse hospitals with limited datasets.
  arXiv  Detail & Related papers  (2024-01-01T10:20:01Z)
• A Knowledge-Driven Cross-view Contrastive Learning for EEG Representation [48.85731427874065]
  This paper proposes a knowledge-driven cross-view contrastive learning framework (KDC2) to extract effective representations from EEG with limited labels. The KDC2 method creates scalp and neural views of EEG signals, simulating the internal and external representation of brain activity. By modeling prior neural knowledge based on neural information consistency theory, the proposed method extracts invariant and complementary neural knowledge to generate combined representations.
  arXiv  Detail & Related papers  (2023-09-21T08:53:51Z)
• DriPP: Driven Point Processes to Model Stimuli Induced Patterns in M/EEG Signals [62.997667081978825]
  We develop a novel statistical point process model-called driven temporal point processes (DriPP) We derive a fast and principled expectation-maximization (EM) algorithm to estimate the parameters of this model. Results on standard MEG datasets demonstrate that our methodology reveals event-related neural responses.
  arXiv  Detail & Related papers  (2021-12-08T13:07:21Z)
• Decoding EEG Brain Activity for Multi-Modal Natural Language Processing [9.35961671939495]
  We present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. For a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines.
  arXiv  Detail & Related papers  (2021-02-17T09:44:21Z)
• Human brain activity for machine attention [8.673635963837532]
  We are the first to exploit neuroscientific data, namely electroencephalography (EEG), to inform a neural attention model about language processing of the human brain. We devise a method for finding such EEG features to supervise machine attention through combining theoretically motivated cropping with random forest tree splits. We apply these features to regularise attention on relation classification and show that EEG is more informative than strong baselines.
  arXiv  Detail & Related papers  (2020-06-09T08:39:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.