RAG-based EEG-to-Text Translation Using Deep Learning and LLMs

• URL: http://arxiv.org/abs/2605.17503v1
• Date: Sun, 17 May 2026 15:31:30 GMT
• Title: RAG-based EEG-to-Text Translation Using Deep Learning and LLMs
• Abstract summary: We propose a retrieval-augmented generation (RAG)-based sentence-level EEG-to-text decoding pipeline.<n>Experiments are conducted on the Zurich Cognitive Language Processing Corpus dataset.
• Score: 3.960347374522111
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The decoding of linguistic information from electroencephalography (EEG) signals remains an extremely challenging problem in brain-computer interface (BCI) research. In particular, sentence-level decoding from EEG is difficult due to the low signal-to-noise ratio of these recordings. Previous studies tackling this problem have typically failed to surpass random baseline performance unless teacher forcing is used during the inference phase. In this work, we propose a retrieval-augmented generation (RAG)-based sentence-level EEG-to-text decoding pipeline that combines an EEG encoder aligned with semantic sentence embeddings, a vector retrieval stage, and a large language model (LLM) to refine retrieved sentences into coherent output. Experiments are conducted on the Zurich Cognitive Language Processing Corpus (ZuCo) dataset, which contains single-trial EEG recordings collected during silent reading. To evaluate whether the system extracts meaningful information from these EEG signals, the results are compared with a random baseline. In nine subjects, the proposed pipeline outperforms the random baseline, achieving a mean cosine similarity of 0.181 +- 0.022 compared to 0.139 +- 0.029 for the baseline, corresponding to a relative improvement of 30.45%. Statistical analysis further confirms that this improvement is significant, following a strict evaluation workflow where inference is performed without access to ground-truth labels.

Related papers

• Membership Inference Attacks against Large Audio Language Models [50.84901010528239]
  We present the first systematic Membership Inference Attack (MIA) evaluation of Large Audio Language Models (LALMs)<n>Using a multi-modal blind baseline based on textual, spectral, and prosodic features, we demonstrate that common speech datasets exhibit near-perfect train/test separability.<n>These findings establish a principled standard for auditing LALMs beyond spurious correlations.
  arXiv  Detail & Related papers  (2026-03-30T12:45:28Z)
• Brain-CLIPLM: Decoding Compressed Semantic Representations in EEG for Language Reconstruction [20.162170862752813]
  We introduce a framework that decomposes EEG-to-text decoding into semantic anchor extraction via contrastive learning and sentence reconstruction.<n>Brain-CLIPLM achieves 67.55% top-5 and 85.00% top-25 sentence retrieval accuracy, significantly outperforming direct decoding baseline.<n>Results suggest EEG-to-text decoding is better framed as recovering compressed semantic content rather than reconstructing full sentences.
  arXiv  Detail & Related papers  (2026-03-23T11:45:51Z)
• Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation [52.51005875755718]
  We focus on EEG-to-text decoding and address its hallucination issue through the lens of posterior collapse.<n>Acknowledging the underlying mismatch in information capacity between EEG and text, we reframe the decoding task as semantic summarization of core meanings.<n>Experiments on the public ZuCo dataset demonstrate that GLIM consistently generates fluent, EEG-grounded sentences.
  arXiv  Detail & Related papers  (2025-05-21T05:29:55Z)
• Decoding EEG Speech Perception with Transformers and VAE-based Data Augmentation [6.405846203953988]
  Decoding speech from electroencephalography (EEG) has the potential to advance brain-computer interfaces (BCIs)<n>EEG-based speech decoding faces major challenges, such as noisy data, limited datasets, and poor performance on complex tasks like speech perception.<n>This study attempts to address these challenges by employing variational autoencoders (VAEs) for EEG data augmentation to improve data quality.
  arXiv  Detail & Related papers  (2025-01-08T08:55:10Z)
• Towards Brain Passage Retrieval -- An Investigation of EEG Query Representations [6.084958172018792]
  Information Retrieval systems rely on users' ability to translate their internal information needs into (text) queries.<n>Current approaches attempting to decode explicit text queries from brain signals have shown limited effectiveness.<n>We propose BPR (Brain Passage Retrieval), a novel framework that eliminates the need for intermediate query translation.
  arXiv  Detail & Related papers  (2024-12-09T17:41:25Z)
• Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings [27.418738450536047]
  We propose a two-step pipeline for converting EEG signals into sentences. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading. We employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder.
  arXiv  Detail & Related papers  (2024-08-08T03:40:25Z)
• Enhancing Electrocardiogram Signal Analysis Using NLP-Inspired Techniques: A Novel Approach with Embedding and Self-Attention [2.7651063843287718]
  We propose a novel ECG analysis technique, based on embedding and self attention, to capture the spatial as well as the temporal dependencies of the ECG data. An accuracy of 91% was achieved with a good F1-score for all the disease classes.
  arXiv  Detail & Related papers  (2024-07-15T12:20:15Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• DGSD: Dynamical Graph Self-Distillation for EEG-Based Auditory Spatial Attention Detection [49.196182908826565]
  Auditory Attention Detection (AAD) aims to detect target speaker from brain signals in a multi-speaker environment. Current approaches primarily rely on traditional convolutional neural network designed for processing Euclidean data like images. This paper proposes a dynamical graph self-distillation (DGSD) approach for AAD, which does not require speech stimuli as input.
  arXiv  Detail & Related papers  (2023-09-07T13:43:46Z)
• Speech Artifact Removal from EEG Recordings of Spoken Word Production with Tensor Decomposition [20.397149635457346]
  Speech artifacts contaminate electroencephalogram (EEG) signals and prevent the inspection of the underlying cognitive processes. To fuel further EEG research with speech production, a method using three-mode tensor decomposition is proposed. In a picture-naming task, we collected raw data with speech artifacts by placing two electrodes near the mouth to record lip EMG.
  arXiv  Detail & Related papers  (2022-06-01T17:10:23Z)
• Generalizing electrocardiogram delineation: training convolutional neural networks with synthetic data augmentation [63.51064808536065]
  Existing databases for ECG delineation are small, being insufficient in size and in the array of pathological conditions they represent. This article delves has two main contributions. First, a pseudo-synthetic data generation algorithm was developed, based in probabilistically composing ECG traces given "pools" of fundamental segments, as cropped from the original databases, and a set of rules for their arrangement into coherent synthetic traces. Second, two novel segmentation-based loss functions have been developed, which attempt at enforcing the prediction of an exact number of independent structures and at producing closer segmentation boundaries by focusing on a reduced number of samples.
  arXiv  Detail & Related papers  (2021-11-25T10:11:41Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.