fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding

• URL: http://arxiv.org/abs/2511.21760v1
• Date: Mon, 24 Nov 2025 20:26:59 GMT
• Title: fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding
• Authors: Yuxiang Wei, Yanteng Zhang, Xi Xiao, Chengxuan Qian, Tianyang Wang, Vince D. Calhoun,
• Abstract summary: We present fMRI-LM, a foundational model that bridges functional MRI (fMRI) and language through a three-stage framework.<n>In Stage 1, we learn a neural tokenizer that maps fMRI into discrete tokens embedded in a language-consistent space.<n>In Stage 2, a pretrained LLM is adapted to jointly model fMRI tokens and text, treating brain activity as a sequence that can be temporally predicted and linguistically described.<n>In Stage 3, we perform multi-task, multi-paradigm instruction tuning to endow fMRI-LM with high-level semantic understanding.
• Score: 30.02799445727658
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in multimodal large language models (LLMs) have enabled unified reasoning across images, audio, and video, but extending such capability to brain imaging remains largely unexplored. Bridging this gap is essential to link neural activity with semantic cognition and to develop cross-modal brain representations. To this end, we present fMRI-LM, a foundational model that bridges functional MRI (fMRI) and language through a three-stage framework. In Stage 1, we learn a neural tokenizer that maps fMRI into discrete tokens embedded in a language-consistent space. In Stage 2, a pretrained LLM is adapted to jointly model fMRI tokens and text, treating brain activity as a sequence that can be temporally predicted and linguistically described. To overcome the lack of natural fMRI-text pairs, we construct a large descriptive corpus that translates diverse imaging-based features into structured textual descriptors, capturing the low-level organization of fMRI signals. In Stage 3, we perform multi-task, multi-paradigm instruction tuning to endow fMRI-LM with high-level semantic understanding, supporting diverse downstream applications. Across various benchmarks, fMRI-LM achieves strong zero-shot and few-shot performance, and adapts efficiently with parameter-efficient tuning (LoRA), establishing a scalable pathway toward a language-aligned, universal model for structural and semantic understanding of fMRI.

Related papers

• OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation [5.3427577036717]
  We introduce OmniMRI, a unified vision-language foundation model designed to generalize across the entire MRI workflow.<n> OmniMRI is trained on a large-scale, heterogeneous corpus curated from 60 public datasets.<n>Results demonstrate OmniMRI's ability to perform diverse tasks within a single architecture.
  arXiv  Detail & Related papers  (2025-08-24T21:11:28Z)
• Perception Activator: An intuitive and portable framework for brain cognitive exploration [19.851643249367108]
  We develop an experimental framework that uses fMRI representations as intervention conditions.<n>We compare both downstream performance and intermediate feature changes on object detection and instance segmentation tasks with and without fMRI information.<n>Our results prove that fMRI contains rich multi-object semantic cues and coarse spatial localization information-elements that current models have yet to fully exploit or integrate.
  arXiv  Detail & Related papers  (2025-07-03T04:46:48Z)
• CodeBrain: Towards Decoupled Interpretability and Multi-Scale Architecture for EEG Foundation Model [52.466542039411515]
  EEG foundation models (EFMs) have emerged to address the scalability issues of task-specific models.<n>We present CodeBrain, a two-stage EFM designed to fill this gap.<n>In the first stage, we introduce the TFDual-Tokenizer, which decouples heterogeneous temporal and frequency EEG signals into discrete tokens.<n>In the second stage, we propose the multi-scale EEGSSM architecture, which combines structured global convolution with sliding window attention.
  arXiv  Detail & Related papers  (2025-06-10T17:20:39Z)
• LLM4Brain: Training a Large Language Model for Brain Video Understanding [9.294352205183726]
  We introduce an LLM-based approach for reconstructing visual-semantic information from fMRI signals elicited by video stimuli. We employ fine-tuning techniques on an fMRI encoder equipped with adaptors to transform brain responses into latent representations aligned with the video stimuli. In particular, we integrate self-supervised domain adaptation methods to enhance the alignment between visual-semantic information and brain responses.
  arXiv  Detail & Related papers  (2024-09-26T15:57:08Z)
• BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Models [0.0]
  This paper proposes BrainChat, a generative framework aimed at rapidly accomplishing semantic information decoding tasks from brain activity. BrainChat implements fMRI question answering and fMRI captioning. BrainChat is highly flexible and can achieve high performance without image data, making it better suited for real-world scenarios with limited data.
  arXiv  Detail & Related papers  (2024-06-10T12:06:15Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation [55.51412454263856]
  This paper proposes to directly modulate the generation process of diffusion models using fMRI signals. By training with about 67,000 fMRI-image pairs from various individuals, our model enjoys superior fMRI-to-image decoding capacity.
  arXiv  Detail & Related papers  (2024-03-27T02:42:52Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding [51.911473457195555]
  BrainCLIP is a task-agnostic fMRI-based brain decoding model. It bridges the modality gap between brain activity, image, and text. BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
  arXiv  Detail & Related papers  (2023-02-25T03:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.