BrainLLM: Generative Language Decoding from Brain Recordings

• URL: http://arxiv.org/abs/2311.09889v6
• Date: Sun, 02 Nov 2025 04:29:03 GMT
• Title: BrainLLM: Generative Language Decoding from Brain Recordings
• Authors: Ziyi Ye, Qingyao Ai, Yiqun Liu, Maarten de Rijke, Min Zhang, Christina Lioma, Tuukka Ruotsalo,
• Abstract summary: We propose a generative language BCI that utilizes the capacity of a large language model and a semantic brain decoder.<n>The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli.<n>Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation.
• Score: 77.66707255697706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating human language through non-invasive brain-computer interfaces (BCIs) has the potential to unlock many applications, such as serving disabled patients and improving communication. Currently, however, generating language via BCIs has been previously successful only within a classification setup for selecting pre-generated sentence continuation candidates with the most likely cortical semantic representation. Inspired by recent research that revealed associations between the brain and the large computational language models, we propose a generative language BCI that utilizes the capacity of a large language model (LLM) jointly with a semantic brain decoder to directly generate language from functional magnetic resonance imaging (fMRI) input. The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli perceived, without prior knowledge of any pre-generated candidates. We compare the language generated from the presented model with a random control, pre-generated language selection approach, and a standard LLM, which generates common coherent text solely based on the next word likelihood according to statistical language training data. The proposed model is found to generate language that is more aligned with semantic stimulus in response to which brain input is sampled. Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation.

Related papers

• A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings [9.11230353886722]
  We propose a joint multilingual, multi-subject and multimodal decoding framework.<n>It maps diverse brain recordings into a unified semantic space defined by a pre-trained multilingual model.<n>The proposed framework can promote linguistic fairness, which is vital for underrepresented languages in BCI applications.
  arXiv  Detail & Related papers  (2025-06-03T04:34:22Z)
• Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models [56.61984030508691]
  We present the first mechanistic interpretability study of language confusion.<n>We show that confusion points (CPs) are central to this phenomenon.<n>We show that editing a small set of critical neurons, identified via comparative analysis with a multilingual-tuned counterpart, substantially mitigates confusion.
  arXiv  Detail & Related papers  (2025-05-22T11:29:17Z)
• Coherent Language Reconstruction from Brain Recordings with Flexible Multi-Modal Input Stimuli [5.589479682782169]
  Decoding thoughts from brain activity offers valuable insights into human cognition and enables promising applications in brain-computer interaction.<n>We propose a unified and flexible framework for reconstructing coherent language from brain recordings elicited by diverse input modalities.
  arXiv  Detail & Related papers  (2025-05-15T14:46:45Z)
• sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment [8.466223794246261]
  We present SSENSE, a contrastive learning framework that projects single-subject stereo-electroencephalography (sEEG) signals into the sentence embedding space of a frozen CLIP model. We evaluate our method on time-aligned sEEG and spoken transcripts from a naturalistic movie-watching dataset.
  arXiv  Detail & Related papers  (2025-04-20T03:01:42Z)
• Generative causal testing to bridge data-driven models and scientific theories in language neuroscience [82.995061475971]
  We present generative causal testing (GCT), a framework for generating concise explanations of language selectivity in the brain. We show that GCT can dissect fine-grained differences between brain areas with similar functional selectivity.
  arXiv  Detail & Related papers  (2024-10-01T15:57:48Z)
• Modelling Multimodal Integration in Human Concept Processing with Vision-Language Models [7.511284868070148]
  We investigate whether integration of visuo-linguistic information leads to representations that are more aligned with human brain activity. Our findings indicate an advantage of multimodal models in predicting human brain activations.
  arXiv  Detail & Related papers  (2024-07-25T10:08:37Z)
• Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network [16.317199232071232]
  Large Language Models (LLMs) have been shown to be effective models of the human language system. In this work, we investigate the key architectural components driving the surprising alignment of untrained models.
  arXiv  Detail & Related papers  (2024-06-21T12:54:03Z)
• Language Reconstruction with Brain Predictive Coding from fMRI Data [28.217967547268216]
  Theory of predictive coding suggests that human brain naturally engages in continuously predicting future word representations. textscPredFT achieves current state-of-the-art decoding performance with a maximum BLEU-1 score of $27.8%$.
  arXiv  Detail & Related papers  (2024-05-19T16:06:02Z)
• Meta predictive learning model of languages in neural circuits [2.5690340428649328]
  We propose a mean-field learning model within the predictive coding framework. Our model reveals that most of the connections become deterministic after learning. Our model provides a starting point to investigate the connection among brain computation, next-token prediction and general intelligence.
  arXiv  Detail & Related papers  (2023-09-08T03:58:05Z)
• Sequential Best-Arm Identification with Application to Brain-Computer Interface [34.87975833920409]
  A brain-computer interface (BCI) is a technology that enables direct communication between the brain and an external device or computer system. An electroencephalogram (EEG) and event-related potential (ERP)-based speller system is a type of BCI that allows users to spell words without using a physical keyboard. We propose a sequential top-two Thompson sampling (STTS) algorithm under the fixed-confidence setting and the fixed-budget setting.
  arXiv  Detail & Related papers  (2023-05-17T18:49:44Z)
• Communication Drives the Emergence of Language Universals in Neural Agents: Evidence from the Word-order/Case-marking Trade-off [3.631024220680066]
  We propose a new Neural-agent Language Learning and Communication framework (NeLLCom) where pairs of speaking and listening agents first learn a miniature language. We succeed in replicating the trade-off with the new framework without hard-coding specific biases in the agents.
  arXiv  Detail & Related papers  (2023-01-30T17:22:33Z)
• LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers [71.76680102779765]
  Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. We propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers.
  arXiv  Detail & Related papers  (2022-11-05T04:03:55Z)
• Dependency-based Mixture Language Models [53.152011258252315]
  We introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention.
  arXiv  Detail & Related papers  (2022-03-19T06:28:30Z)
• Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification [78.120927891455]
  State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines.
  arXiv  Detail & Related papers  (2021-12-05T21:57:22Z)
• Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects [82.81964713263483]
  A popular approach to decompose the neural bases of language consists in correlating, across individuals, the brain responses to different stimuli. Here, we show that a model-based approach can reach equivalent results within subjects exposed to natural stimuli.
  arXiv  Detail & Related papers  (2021-10-12T15:30:21Z)
• Towards Zero-shot Language Modeling [90.80124496312274]
  We construct a neural model that is inductively biased towards learning human languages. We infer this distribution from a sample of typologically diverse training languages. We harness additional language-specific side information as distant supervision for held-out languages.
  arXiv  Detail & Related papers  (2021-08-06T23:49:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.