Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging

• URL: http://arxiv.org/abs/2606.14957v2
• Date: Thu, 18 Jun 2026 14:47:30 GMT
• Title: Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging
• Abstract summary: We introduce Neuro-JEPA, a sparse multimodal neuroimaging foundation model that encodes brain MRI across core T1w, T2w, and fluid-suppressed FLAIR imaging (FLAIR)<n>Neuro-JEPA was pretrained on 1,551,862 scans from 428,647 studies after modality-specific preprocessing with data curation across three core structural brain MRI sequences.<n>We evaluated the learned representations across clinical and research settings, including 25 tasks from three health systems.
• Score: 5.655046812740412
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Brain MRIs are routinely acquired as multiple complementary sequences with unique contrast weighting, including T1-weighed imaging (T1w) anatomic and fluid-sensitive T2-weighted (T2w) contrasts. However, methods for learning unified representations across the multitude of MRI contrast mechanisms at health-system scale are lacking. In this study, we introduce Neuro-JEPA, a sparse multimodal neuroimaging foundation model that combines a latent predictive objective with a Mixture-of-Experts architecture to encode brain MRI across core T1w, T2w, and fluid-suppressed FLAIR imaging (FLAIR). We further provide a systematic methodological study of architectural, masking, objective, and sparsity design choices beneficial for robust neuroimaging multimodal representation learning. Neuro-JEPA was pretrained on 1,551,862 scans from 428,647 studies after modality-specific preprocessing with data curation across three core structural brain MRI sequences. We evaluated the learned representations across clinical and research settings, including 25 tasks from three health systems: NYU Langone, NYU Long Island, and Massachusetts General Hospital, and 22 tasks from 12 public datasets, covering unimodal, multimodal and cross-domain evaluation configurations. Across these benchmarks, existing neuroimaging foundation models showed inconsistent gains over a simple convolutional neural network (CNN) baseline, whereas Neuro-JEPA achieved stronger and more consistent performance across all evaluated settings. These results establish a scalable methodological framework for multimodal neuroimaging representation learning and highlight the need for foundation model evaluation protocols that include simple baselines, clinically heterogeneous cohorts and controlled multimodal comparisons.

Related papers

• Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI [15.657081811594418]
  We propose a modality-aware 3D vector-quantized VAE (VQ-VAE) for reconstructing multi-modal brain MRIs.<n>NeuroQuant achieves superior reconstruction fidelity compared to existing VAEs.
  arXiv  Detail & Related papers  (2026-04-06T21:07:34Z)
• Towards a general-purpose foundation model for fMRI analysis [58.06455456423138]
  We introduce NeuroSTORM, a framework that learns from 4D fMRI volumes and enables efficient knowledge transfer across diverse applications.<n>NeuroSTORM is pre-trained on 28.65 million fMRI frames (>9,000 hours) from over 50,000 subjects across multiple centers and ages 5 to 100.<n>It outperforms existing methods across five tasks: age/gender prediction, phenotype prediction, disease diagnosis, fMRI-to-image retrieval, and task-based fMRI.
  arXiv  Detail & Related papers  (2025-06-11T23:51:01Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation [56.34634121544929]
  In this study, we first construct the brain-effective network via the dynamic causal model. We then introduce an interpretable graph learning framework termed Spatio-Temporal Embedding ODE (STE-ODE) This framework incorporates specifically designed directed node embedding layers, aiming at capturing the dynamic interplay between structural and effective networks.
  arXiv  Detail & Related papers  (2024-05-21T20:37:07Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation [55.51412454263856]
  This paper proposes to directly modulate the generation process of diffusion models using fMRI signals. By training with about 67,000 fMRI-image pairs from various individuals, our model enjoys superior fMRI-to-image decoding capacity.
  arXiv  Detail & Related papers  (2024-03-27T02:42:52Z)
• SF2Former: Amyotrophic Lateral Sclerosis Identification From Multi-center MRI Data Using Spatial and Frequency Fusion Transformer [3.408266725482757]
  Amyotrophic Lateral Sclerosis (ALS) is a complex neurodegenerative disorder involving motor neuron degeneration. Deep learning has turned into a prominent class of machine learning programs in computer vision. This study introduces a framework named SF2Former that leverages vision transformer architecture's power to distinguish the ALS subjects from the control group.
  arXiv  Detail & Related papers  (2023-02-21T18:16:20Z)
• Model-Guided Multi-Contrast Deep Unfolding Network for MRI Super-resolution Reconstruction [68.80715727288514]
  We show how to unfold an iterative MGDUN algorithm into a novel model-guided deep unfolding network by taking the MRI observation matrix. In this paper, we propose a novel Model-Guided interpretable Deep Unfolding Network (MGDUN) for medical image SR reconstruction.
  arXiv  Detail & Related papers  (2022-09-15T03:58:30Z)
• A Neural Ordinary Differential Equation Model for Visualizing Deep Neural Network Behaviors in Multi-Parametric MRI based Glioma Segmentation [3.1435638364138105]
  We develop a neural ordinary differential equation (ODE) model for visualizing deep neural network (DNN) during multi-parametric MRI (mp-MRI) based glioma segmentation. All neural ODE models successfully illustrated image dynamics as expected.
  arXiv  Detail & Related papers  (2022-03-01T17:16:41Z)
• Neural Architecture Search for Gliomas Segmentation on Multimodal Magnetic Resonance Imaging [2.66512000865131]
  We propose a neural architecture search (NAS) based solution to brain tumor segmentation tasks on multimodal MRI scans. The developed solution also integrates normalization and patching strategies tailored for brain MRI processing.
  arXiv  Detail & Related papers  (2020-05-13T14:32:00Z)
• Mapping individual differences in cortical architecture using multi-view representation learning [0.0]
  We introduce a novel machine learning method which allows combining the activation-and connectivity-based information respectively measured through task-fMRI and resting-state fMRI. It combines a multi-view deep autoencoder which is designed to fuse the two fMRI modalities into a joint representation space within which a predictive model is trained to guess a scalar score that characterizes the patient.
  arXiv  Detail & Related papers  (2020-04-01T09:01:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.