Brain-CLIPLM: Decoding Compressed Semantic Representations in EEG for Language Reconstruction

• URL: http://arxiv.org/abs/2604.16370v1
• Date: Mon, 23 Mar 2026 11:45:51 GMT
• Title: Brain-CLIPLM: Decoding Compressed Semantic Representations in EEG for Language Reconstruction
• Abstract summary: We introduce a framework that decomposes EEG-to-text decoding into semantic anchor extraction via contrastive learning and sentence reconstruction.<n>Brain-CLIPLM achieves 67.55% top-5 and 85.00% top-25 sentence retrieval accuracy, significantly outperforming direct decoding baseline.<n>Results suggest EEG-to-text decoding is better framed as recovering compressed semantic content rather than reconstructing full sentences.
• Score: 20.162170862752813
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decoding natural language from non-invasive electroencephalography (EEG) remains fundamentally limited by low signal-to-noise ratio and restricted information bandwidth. This raises a fundamental question regarding whether sentence-level linguistic structure can be reliably recovered from such signals. In this work, we suggest that this assumption may not hold under realistic information constraints, and instead propose a semantic compression hypothesis in which EEG signals encode a compressed set of semantic anchors rather than full linguistic structure. Under our new perspective, direct sentence reconstruction becomes an overparameterized objective relative to the intrinsic information capacity of EEG. To address this mismatch, we introduce Brain-CLIPLM, a two-stage framework that decomposes EEG-to-text decoding into semantic anchor extraction via contrastive learning and sentence reconstruction using a retrieval-grounded large language model (LLM) with Chain-of-Thought (CoT) reasoning, following a granularity matching principle that aligns decoding complexity with neural information capacity. Evaluated on the Zurich Cognitive Language Processing Corpus, Brain-CLIPLM achieves 67.55\% top-5 and 85.00\% top-25 sentence retrieval accuracy, significantly outperforming direct decoding baseline, while cross-subject evaluation confirms robust generalization. Control analyses, including permutation testing, further demonstrate that EEG-derived representations carry sentence-specific information beyond language model priors. These results suggest that EEG-to-text decoding is better framed as recovering compressed semantic content rather than reconstructing full sentences, providing a biologically grounded and data-efficient pathway for non-invasive brain-computer interfaces.

Related papers

• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• RAG-based EEG-to-Text Translation Using Deep Learning and LLMs [3.960347374522111]
  We propose a retrieval-augmented generation (RAG)-based sentence-level EEG-to-text decoding pipeline.<n>Experiments are conducted on the Zurich Cognitive Language Processing Corpus dataset.
  arXiv  Detail & Related papers  (2026-05-17T15:31:30Z)
• SENSE: Efficient EEG-to-Text via Privacy-Preserving Semantic Retrieval [6.012652594597545]
  Decoding brain activity into natural language is a major challenge in AI with important applications in assistive communication, neurotechnology, and human-computer interaction.<n>Most existing Brain-Computer Interface (BCI) approaches rely on memory-intensive fine-tuning of Large Language Models (LLMs) or encoder-decoder models on raw EEG signals.<n>We introduce SENSE, a lightweight and privacy-preserving framework that translates non-invasive electroencephalography (EEG) into text without LLM fine-tuning.
  arXiv  Detail & Related papers  (2026-03-17T19:59:41Z)
• Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation [52.51005875755718]
  We focus on EEG-to-text decoding and address its hallucination issue through the lens of posterior collapse.<n>Acknowledging the underlying mismatch in information capacity between EEG and text, we reframe the decoding task as semantic summarization of core meanings.<n>Experiments on the public ZuCo dataset demonstrate that GLIM consistently generates fluent, EEG-grounded sentences.
  arXiv  Detail & Related papers  (2025-05-21T05:29:55Z)
• sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment [8.466223794246261]
  We present SSENSE, a contrastive learning framework that projects single-subject stereo-electroencephalography (sEEG) signals into the sentence embedding space of a frozen CLIP model.<n>We evaluate our method on time-aligned sEEG and spoken transcripts from a naturalistic movie-watching dataset.
  arXiv  Detail & Related papers  (2025-04-20T03:01:42Z)
• Towards Brain Passage Retrieval -- An Investigation of EEG Query Representations [6.084958172018792]
  Information Retrieval systems rely on users' ability to translate their internal information needs into (text) queries.<n>Current approaches attempting to decode explicit text queries from brain signals have shown limited effectiveness.<n>We propose BPR (Brain Passage Retrieval), a novel framework that eliminates the need for intermediate query translation.
  arXiv  Detail & Related papers  (2024-12-09T17:41:25Z)
• Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings [27.418738450536047]
  We propose a two-step pipeline for converting EEG signals into sentences. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading. We employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder.
  arXiv  Detail & Related papers  (2024-08-08T03:40:25Z)
• Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder [69.7813498468116]
  We propose Contrastive EEG-Text Masked Autoencoder (CET-MAE), a novel model that orchestrates compound self-supervised learning across and within EEG and text. We also develop a framework called E2T-PTR (EEG-to-Text decoding using Pretrained Transferable Representations) to decode text from EEG sequences.
  arXiv  Detail & Related papers  (2024-02-27T11:45:21Z)
• BrainLLM: Generative Language Decoding from Brain Recordings [77.66707255697706]
  We propose a generative language BCI that utilizes the capacity of a large language model and a semantic brain decoder.<n>The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli.<n>Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation.
  arXiv  Detail & Related papers  (2023-11-16T13:37:21Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification [78.120927891455]
  State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines.
  arXiv  Detail & Related papers  (2021-12-05T21:57:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.