Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG

• URL: http://arxiv.org/abs/2506.00381v1
• Date: Sat, 31 May 2025 04:17:19 GMT
• Title: Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG
• Authors: Siavash Shams, Richard Antonello, Gavin Mischler, Stephan Bickel, Ashesh Mehta, Nima Mesgarani,
• Abstract summary: We introduce Neuro2Semantic, a novel framework that reconstructs the semantic content of perceived speech from intracranial EEG (iEEG) recordings.<n>Our approach consists of two phases: first, an LSTM-based adapter aligns neural signals with pre-trained text embeddings; second, a corrector module generates continuous, natural text directly from these aligned embeddings.<n>Neuro2Semantic achieves strong performance with as little as 30 minutes of neural data, outperforming a recent state-of-the-art method in low-data settings.
• Score: 11.531598524209969
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decoding continuous language from neural signals remains a significant challenge in the intersection of neuroscience and artificial intelligence. We introduce Neuro2Semantic, a novel framework that reconstructs the semantic content of perceived speech from intracranial EEG (iEEG) recordings. Our approach consists of two phases: first, an LSTM-based adapter aligns neural signals with pre-trained text embeddings; second, a corrector module generates continuous, natural text directly from these aligned embeddings. This flexible method overcomes the limitations of previous decoding approaches and enables unconstrained text generation. Neuro2Semantic achieves strong performance with as little as 30 minutes of neural data, outperforming a recent state-of-the-art method in low-data settings. These results highlight the potential for practical applications in brain-computer interfaces and neural decoding technologies.

Related papers

• Neural-Driven Image Editing [51.11173675034121]
  Traditional image editing relies on manual prompting, making it labor-intensive and inaccessible to individuals with limited motor control or language abilities.<n>We propose LoongX, a hands-free image editing approach driven by neurophysiological signals.<n>LoongX utilizes state-of-the-art diffusion models trained on a comprehensive dataset of 23,928 image editing pairs.
  arXiv  Detail & Related papers  (2025-07-07T18:31:50Z)
• Towards Unified Neural Decoding with Brain Functional Network Modeling [34.13766828046489]
  We present Multi-individual Brain Region-Aggregated Network (MIBRAIN), a neural decoding framework.<n>MIBRAIN constructs a whole functional brain network model by integrating intracranial neurophysiological recordings across multiple individuals.<n>Our framework paves the way for robust neural decoding across individuals and offers insights for practical clinical applications.
  arXiv  Detail & Related papers  (2025-05-30T12:10:37Z)
• BrainStratify: Coarse-to-Fine Disentanglement of Intracranial Neural Dynamics [8.36470471250669]
  Decoding speech directly from neural activity is a central goal in brain-computer interface (BCI) research.<n>In recent years, exciting advances have been made through the growing use of intracranial field potential recordings, such as stereo-ElectroEncephaloGraphy (sEEG) and ElectroCorticoGraphy (ECoG)<n>These neural signals capture rich population-level activity but present key challenges: (i) task-relevant neural signals are sparsely distributed across sEEG electrodes, and (ii) they are often entangled with task-irrelevant neural signals in both sEEG and ECo
  arXiv  Detail & Related papers  (2025-05-26T19:36:39Z)
• sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment [8.466223794246261]
  We present SSENSE, a contrastive learning framework that projects single-subject stereo-electroencephalography (sEEG) signals into the sentence embedding space of a frozen CLIP model.<n>We evaluate our method on time-aligned sEEG and spoken transcripts from a naturalistic movie-watching dataset.
  arXiv  Detail & Related papers  (2025-04-20T03:01:42Z)
• neuro2voc: Decoding Vocalizations from Neural Activity [3.1913357260723303]
  This master project investigates experimental methods for decoding zebra finch motor outputs.<n>XGBoost with SHAP analysis trained on spike rates revealed neuronal interaction patterns crucial for syllable classification.<n>A combined contrastive learning-VAE framework successfully generated spectrograms from binned neural data.
  arXiv  Detail & Related papers  (2025-02-02T11:09:31Z)
• Neural Spelling: A Spell-Based BCI System for Language Neural Decoding [25.721498096893427]
  We propose a novel non-invasive EEG-based BCI system with Curriculum-based Neural Spelling Framework.<n>It recognizes all 26 alphabet letters by decoding neural signals associated with handwriting first, and then apply a Generative AI (GenAI) to enhance spell-based neural language decoding tasks.
  arXiv  Detail & Related papers  (2025-01-29T08:57:51Z)
• NeuGPT: Unified multi-modal Neural GPT [48.70587003475798]
  NeuGPT is a groundbreaking multi-modal language generation model designed to harmonize the fragmented landscape of neural recording research. Our model mainly focus on brain-to-text decoding, improving SOTA from 6.94 to 12.92 on BLEU-1 and 6.93 to 13.06 on ROUGE-1F. It can also simulate brain signals, thereby serving as a novel neural interface.
  arXiv  Detail & Related papers  (2024-10-28T10:53:22Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• Surrogate Gradient Spiking Neural Networks as Encoders for Large Vocabulary Continuous Speech Recognition [91.39701446828144]
  We show that spiking neural networks can be trained like standard recurrent neural networks using the surrogate gradient method. They have shown promising results on speech command recognition tasks. In contrast to their recurrent non-spiking counterparts, they show robustness to exploding gradient problems without the need to use gates.
  arXiv  Detail & Related papers  (2022-12-01T12:36:26Z)
• Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification [78.120927891455]
  State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines.
  arXiv  Detail & Related papers  (2021-12-05T21:57:22Z)
• Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI [2.0178765779788495]
  Decoding behavior, perception, or cognitive state directly from neural signals has applications in brain-computer interface research. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks.
  arXiv  Detail & Related papers  (2020-05-19T18:10:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.