A Modality-agnostic Multi-task Foundation Model for Human Brain Imaging

• URL: http://arxiv.org/abs/2509.00549v1
• Date: Sat, 30 Aug 2025 16:15:32 GMT
• Title: A Modality-agnostic Multi-task Foundation Model for Human Brain Imaging
• Authors: Peirong Liu, Oula Puonti, Xiaoling Hu, Karthik Gopinath, Annabel Sorby-Adams, Daniel C. Alexander, W. Taylor Kimberly, Juan E. Iglesias,
• Abstract summary: We introduce BrainFM, a modality-agnostic, multi-task vision foundation model for human brain imaging.<n>BrainFM is resilient to the appearance of acquired images.<n>It can be directly applied to five fundamental brain imaging tasks.
• Score: 12.710492824928338
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent learning-based approaches have made astonishing advances in calibrated medical imaging like computerized tomography (CT), yet they struggle to generalize in uncalibrated modalities -- notably magnetic resonance (MR) imaging, where performance is highly sensitive to the differences in MR contrast, resolution, and orientation. This prevents broad applicability to diverse real-world clinical protocols. Here we introduce BrainFM, a modality-agnostic, multi-task vision foundation model for human brain imaging. With the proposed "mild-to-severe" intra-subject generation and "real-synth" mix-up training strategy, BrainFM is resilient to the appearance of acquired images (e.g., modality, contrast, deformation, resolution, artifacts), and can be directly applied to five fundamental brain imaging tasks, including image synthesis for CT and T1w/T2w/FLAIR MRI, anatomy segmentation, scalp-to-cortical distance, bias field estimation, and registration. We evaluate the efficacy of BrainFM on eleven public datasets, and demonstrate its robustness and effectiveness across all tasks and input modalities. Code is available at https://github.com/jhuldr/BrainFM.

Related papers

• Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training [4.672822120456554]
  Most segmentation models for multimodal brain MRI are restricted to fixed modalities.<n>Some models generalize to unseen modalities but may lose modality-specific information.<n>This work aims to develop a model that can perform inference on data that contain image modalities unseen during training.
  arXiv  Detail & Related papers  (2025-09-11T09:25:30Z)
• Bidirectional Brain Image Translation using Transfer Learning from Generic Pre-trained Models [0.0]
  In the medical domain, where obtaining labeled medical images is labor-intensive and expensive, addressing data scarcity is a major challenge.<n>Recent studies propose using transfer learning to overcome this issue.<n>In this work, transfer learning was applied to the task of MR-CT image translation and vice versa using 18 pre-trained non-medical models.
  arXiv  Detail & Related papers  (2025-01-21T20:30:15Z)
• Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion [0.11249583407496218]
  This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across multimodal representationsof brain activity by leveraging contrastive learning. We used electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. Our framework's capabilities are demonstrated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities.
  arXiv  Detail & Related papers  (2024-11-14T12:27:27Z)
• Multi-modal Vision Pre-training for Medical Image Analysis [11.569448567735435]
  Self-supervised learning has greatly facilitated medical image analysis by suppressing the training data requirement for real-world applications.<n>We conduct a novel multi-modal image pre-training with three proxy tasks to facilitate the learning of cross-modality representations and correlations.<n>Our method is reported in comparison to state-of-the-art pre-training methods, with Dice Score improvement of 0.28%-14.47% across six segmentation benchmarks and a consistent accuracy boost of 0.65%-18.07% in four individual image classification tasks.
  arXiv  Detail & Related papers  (2024-10-14T15:12:16Z)
• Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation [51.28453192441364]
  Multimodal brain magnetic resonance (MR) imaging is indispensable in neuroscience and neurology. Current MR image synthesis approaches are typically trained on independent datasets for specific tasks. We present TUMSyn, a Text-guided Universal MR image Synthesis model, which can flexibly generate brain MR images.
  arXiv  Detail & Related papers  (2024-09-25T11:14:47Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity [60.983327742457995]
  Reconstructing the viewed images from human brain activity bridges human and computer vision through the Brain-Computer Interface. We devise Psychometry, an omnifit model for reconstructing images from functional Magnetic Resonance Imaging (fMRI) obtained from different subjects.
  arXiv  Detail & Related papers  (2024-03-29T07:16:34Z)
• Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser [12.402947207350394]
  We propose a transformer-based modality infuser designed to synthesize multimodal brain MR images. In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features. We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities.
  arXiv  Detail & Related papers  (2024-02-01T06:34:35Z)
• Brain-ID: Learning Contrast-agnostic Anatomical Representations for Brain Imaging [11.06907516321673]
  We introduce Brain-ID, an anatomical representation learning model for brain imaging. With the proposed "mild-to-severe" intrasubject generation, Brain-ID is robust to the subject-specific brain anatomy. We present new metrics to validate the intra- and inter-subject robustness, and evaluate their performance on four downstream applications.
  arXiv  Detail & Related papers  (2023-11-28T16:16:10Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding [51.911473457195555]
  BrainCLIP is a task-agnostic fMRI-based brain decoding model. It bridges the modality gap between brain activity, image, and text. BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
  arXiv  Detail & Related papers  (2023-02-25T03:28:54Z)
• Cross-Modality Deep Feature Learning for Brain Tumor Segmentation [158.8192041981564]
  This paper proposes a novel cross-modality deep feature learning framework to segment brain tumors from the multi-modality MRI data. The core idea is to mine rich patterns across the multi-modality data to make up for the insufficient data scale. Comprehensive experiments are conducted on the BraTS benchmarks, which show that the proposed cross-modality deep feature learning framework can effectively improve the brain tumor segmentation performance.
  arXiv  Detail & Related papers  (2022-01-07T07:46:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.