MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data

• URL: http://arxiv.org/abs/2606.20696v1
• Date: Mon, 15 Jun 2026 08:30:03 GMT
• Title: MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data
• Abstract summary: We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals.<n> Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines.
• Score: 4.280521231404364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decoding inner speech from non-invasive brain signals remains a fundamental challenge due to the absence of overt linguistic output, limited training data, and large inter-subject variability. Existing brain-to-text approaches often rely on task-specific decoder fine-tuning, which restricts scalability and complicates adaptation to new participants. We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals without modifying the underlying language model. The first stage learns a subject-specific neural-semantic alignment that maps fMRI activity into a shared multimodal semantic space, extracting a latent semantic sketch of the internally generated sentence. The second stage integrates this sketch with visual context to prompt a frozen multimodal language model for free-form generation. Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines. We further show that the learned semantic-to-language projection can generalize across subjects, enabling effective decoding when paired with subject-specific neural alignment. These results indicate that neural signals modulate semantic content beyond image-driven priors, supporting a scalable and modular direction for brain-to-text decoding.

Related papers

• BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language [65.1211456110319]
  BrainJanus is the first unified brain model that integrates brain, vision, and language within a single framework.<n>Our framework exhibits zero-shot generalization preserves and interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm.
  arXiv  Detail & Related papers  (2026-06-29T14:02:15Z)
• fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignment [47.45203641583922]
  We introduce a novel approach, textbffMRI2GES, that allows training of fMRI-to-gesture reconstruction networks on unpaired data.<n>We show that our proposed method can reconstruct expressive gestures directly from fMRI recordings.
  arXiv  Detail & Related papers  (2025-12-01T02:09:44Z)
• fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding [30.02799445727658]
  We present fMRI-LM, a foundational model that bridges functional MRI (fMRI) and language through a three-stage framework.<n>In Stage 1, we learn a neural tokenizer that maps fMRI into discrete tokens embedded in a language-consistent space.<n>In Stage 2, a pretrained LLM is adapted to jointly model fMRI tokens and text, treating brain activity as a sequence that can be temporally predicted and linguistically described.<n>In Stage 3, we perform multi-task, multi-paradigm instruction tuning to endow fMRI-LM with high-level semantic understanding.
  arXiv  Detail & Related papers  (2025-11-24T20:26:59Z)
• Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG [11.531598524209969]
  We introduce Neuro2Semantic, a novel framework that reconstructs the semantic content of perceived speech from intracranial EEG (iEEG) recordings.<n>Our approach consists of two phases: first, an LSTM-based adapter aligns neural signals with pre-trained text embeddings; second, a corrector module generates continuous, natural text directly from these aligned embeddings.<n>Neuro2Semantic achieves strong performance with as little as 30 minutes of neural data, outperforming a recent state-of-the-art method in low-data settings.
  arXiv  Detail & Related papers  (2025-05-31T04:17:19Z)
• sEEG-based Encoding for Sentence Retrieval: A Contrastive Learning Approach to Brain-Language Alignment [8.466223794246261]
  We present SSENSE, a contrastive learning framework that projects single-subject stereo-electroencephalography (sEEG) signals into the sentence embedding space of a frozen CLIP model.<n>We evaluate our method on time-aligned sEEG and spoken transcripts from a naturalistic movie-watching dataset.
  arXiv  Detail & Related papers  (2025-04-20T03:01:42Z)
• BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Models [0.0]
  This paper proposes BrainChat, a generative framework aimed at rapidly accomplishing semantic information decoding tasks from brain activity. BrainChat implements fMRI question answering and fMRI captioning. BrainChat is highly flexible and can achieve high performance without image data, making it better suited for real-world scenarios with limited data.
  arXiv  Detail & Related papers  (2024-06-10T12:06:15Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• BrainLLM: Generative Language Decoding from Brain Recordings [77.66707255697706]
  We propose a generative language BCI that utilizes the capacity of a large language model and a semantic brain decoder.<n>The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli.<n>Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation.
  arXiv  Detail & Related papers  (2023-11-16T13:37:21Z)
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding [61.02342238771685]
  Spoken language understanding requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. Various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. We propose a novel semi-supervised learning framework, SPLAT, to jointly pre-train the speech and language modules.
  arXiv  Detail & Related papers  (2020-10-05T19:29:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.