A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings

• URL: http://arxiv.org/abs/2506.03214v1
• Date: Tue, 03 Jun 2025 04:34:22 GMT
• Title: A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings
• Authors: Yi Guo, Yihang Dong, Michael Kwok-Po Ng, Shuqiang Wang,
• Abstract summary: We propose a joint multilingual, multi-subject and multimodal decoding framework.<n>It maps diverse brain recordings into a unified semantic space defined by a pre-trained multilingual model.<n>The proposed framework can promote linguistic fairness, which is vital for underrepresented languages in BCI applications.
• Score: 9.11230353886722
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Brain-computer interfaces (BCIs) with speech decoding from brain recordings have broad application potential in fields such as clinical rehabilitation and cognitive neuroscience. However, current decoding methods remain limited to single-language, single-subject, and single neuroimaging modality settings, restricting their clinical applicability and generalizability. Here we propose a joint multilingual, multi-subject and multimodal decoding framework. It maps diverse brain recordings into a unified semantic space defined by a pre-trained multilingual model (PMM), enabling decoding across multiple languages, multiple subjects and multiple neuroimaging modalities. The proposed framework is validated using non-invasive brain recordings from 159 participants across four languages. Experimental results show that it exhibits strong generalization across multilingual, multi-subject, and multimodal settings. More importantly, the proposed framework can promote linguistic fairness, which is vital for underrepresented languages in BCI applications. The unified semantic space enables cross-lingual mapping enhancement, allowing the framework to boost the decoding performance of underrepresented languages, thereby promoting linguistic fairness. Overall, the proposed framework establishes a new potential paradigm for brain decoding, opening new paths for broader applications of BCI.

Related papers

• How does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective [64.79894853375478]
  We propose a new finer-grained neuron identification algorithm, which detects language neurons(including language-specific neurons and language-related neurons) and language-agnostic neurons.<n>Based on the distributional characteristics of different types of neurons, we divide the LLMs' internal process for multilingual inference into four parts.<n>We systematically analyze the models before and after alignment with a focus on different types of neurons.
  arXiv  Detail & Related papers  (2025-05-27T17:59:52Z)
• Coherent Language Reconstruction from Brain Recordings with Flexible Multi-Modal Input Stimuli [5.589479682782169]
  Decoding thoughts from brain activity offers valuable insights into human cognition and enables promising applications in brain-computer interaction.<n>We propose a unified and flexible framework for reconstructing coherent language from brain recordings elicited by diverse input modalities.
  arXiv  Detail & Related papers  (2025-05-15T14:46:45Z)
• Multilingual Knowledge Editing with Language-Agnostic Factual Neurons [98.73585104789217]
  Same factual knowledge in different languages generally activates a shared set of neurons, which we call language-agnostic factual neurons (LAFNs)<n>These neurons represent the same factual knowledge shared across languages and imply the semantic connections among multilingual knowledge.<n>We propose a new MKE method by Locating and Updating Language-Agnostic Factual Neurons (LU-LAFNs) to edit multilingual knowledge simultaneously.
  arXiv  Detail & Related papers  (2024-06-24T08:06:56Z)
• Decoding Continuous Character-based Language from Non-invasive Brain Recordings [33.11373366800627]
  We propose a novel approach to decoding continuous language from single-trial non-invasive fMRI recordings. A character-based decoder is designed for the semantic reconstruction of continuous language characterized by inherent character structures. The ability to decode continuous language from single trials across subjects demonstrates the promising applications of non-invasive language brain-computer interfaces.
  arXiv  Detail & Related papers  (2024-03-17T12:12:33Z)
• How do Large Language Models Handle Multilingualism? [81.15060972112563]
  This study explores how large language models (LLMs) handle multilingualism. LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures.
  arXiv  Detail & Related papers  (2024-02-29T02:55:26Z)
• Towards Building Multilingual Language Model for Medicine [54.1382395897071]
  We construct a multilingual medical corpus, containing approximately 25.5B tokens encompassing 6 main languages. We propose a multilingual medical multi-choice question-answering benchmark with rationale, termed as MMedBench. Our final model, MMed-Llama 3, with only 8B parameters, achieves superior performance compared to all other open-source models on both MMedBench and English benchmarks.
  arXiv  Detail & Related papers  (2024-02-21T17:47:20Z)
• Language Generation from Brain Recordings [68.97414452707103]
  We propose a generative language BCI that utilizes the capacity of a large language model and a semantic brain decoder. The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli. Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation.
  arXiv  Detail & Related papers  (2023-11-16T13:37:21Z)
• LAE: Language-Aware Encoder for Monolingual and Multilingual ASR [87.74794847245536]
  A novel language-aware encoder (LAE) architecture is proposed to handle both situations by disentangling language-specific information. Experiments conducted on Mandarin-English code-switched speech suggest that the proposed LAE is capable of discriminating different languages in frame-level.
  arXiv  Detail & Related papers  (2022-06-05T04:03:12Z)
• Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding [90.87454350016121]
  We develop novel code-switching schemes to generate hard negative examples for contrastive learning at all levels. We develop a label-aware joint model to leverage label semantics for cross-lingual knowledge transfer.
  arXiv  Detail & Related papers  (2022-05-07T13:44:28Z)
• Multi-lingual agents through multi-headed neural networks [0.0]
  This paper focuses on cooperative Multi-Agent Reinforcement Learning. In this context, multiple distinct and incompatible languages can emerge. We take inspiration from the Continual Learning literature and equip our agents with multi-headed neural networks which enable our agents to be multi-lingual.
  arXiv  Detail & Related papers  (2021-11-22T11:39:42Z)
• Crossing the Conversational Chasm: A Primer on Multilingual Task-Oriented Dialogue Systems [51.328224222640614]
  Current state-of-the-art ToD models based on large pretrained neural language models are data hungry. Data acquisition for ToD use cases is expensive and tedious.
  arXiv  Detail & Related papers  (2021-04-17T15:19:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.