MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding

• URL: http://arxiv.org/abs/2505.15946v2
• Date: Mon, 26 May 2025 03:02:00 GMT
• Title: MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding
• Authors: Yuxiang Wei, Yanteng Zhang, Xi Xiao, Tianyang Wang, Xiao Wang, Vince D. Calhoun,
• Abstract summary: MoRE-Brain is a neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction.<n>It employs a hierarchical Mixture-of-Experts architecture where distinct experts process fMRI signals.<n>MoRE-Brain marks a substantial advance towards more generalizable and interpretable fMRI-based visual decoding.
• Score: 24.771496672135395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decoding visual experiences from fMRI offers a powerful avenue to understand human perception and develop advanced brain-computer interfaces. However, current progress often prioritizes maximizing reconstruction fidelity while overlooking interpretability, an essential aspect for deriving neuroscientific insight. To address this gap, we propose MoRE-Brain, a neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction. MoRE-Brain uniquely employs a hierarchical Mixture-of-Experts architecture where distinct experts process fMRI signals from functionally related voxel groups, mimicking specialized brain networks. The experts are first trained to encode fMRI into the frozen CLIP space. A finetuned diffusion model then synthesizes images, guided by expert outputs through a novel dual-stage routing mechanism that dynamically weighs expert contributions across the diffusion process. MoRE-Brain offers three main advancements: First, it introduces a novel Mixture-of-Experts architecture grounded in brain network principles for neuro-decoding. Second, it achieves efficient cross-subject generalization by sharing core expert networks while adapting only subject-specific routers. Third, it provides enhanced mechanistic insight, as the explicit routing reveals precisely how different modeled brain regions shape the semantic and spatial attributes of the reconstructed image. Extensive experiments validate MoRE-Brain's high reconstruction fidelity, with bottleneck analyses further demonstrating its effective utilization of fMRI signals, distinguishing genuine neural decoding from over-reliance on generative priors. Consequently, MoRE-Brain marks a substantial advance towards more generalizable and interpretable fMRI-based visual decoding. Code will be publicly available soon: https://github.com/yuxiangwei0808/MoRE-Brain.

Related papers

• MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data [64.92867794764247]
  MindAligner is a framework for cross-subject brain decoding from limited fMRI data.<n>Brain Transfer Matrix (BTM) projects the brain signals of an arbitrary new subject to one of the known subjects.<n>Brain Functional Alignment module is proposed to perform soft cross-subject brain alignment under different visual stimuli.
  arXiv  Detail & Related papers  (2025-02-07T16:01:59Z)
• UniBrain: A Unified Model for Cross-Subject Brain Decoding [22.49964298783508]
  We present UniBrain, a unified brain decoding model that requires no subject-specific parameters.<n>Our approach includes a group-based extractor to handle variable fMRI signal lengths, a mutual assistance embedder to capture cross-subject commonalities, and a bilevel feature alignment scheme for extracting subject-invariant features.<n>We validate our UniBrain on the brain decoding benchmark, achieving comparable performance to current state-of-the-art subject-specific models with extremely fewer parameters.
  arXiv  Detail & Related papers  (2024-12-27T07:03:47Z)
• BrainMAP: Learning Multiple Activation Pathways in Brain Networks [77.15180533984947]
  We introduce a novel framework BrainMAP to learn Multiple Activation Pathways in Brain networks.<n>Our framework enables explanatory analyses of crucial brain regions involved in tasks.
  arXiv  Detail & Related papers  (2024-12-23T09:13:35Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction [8.63068449082585]
  Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition. Our framework integrates 3D brain structures with visual semantics using a Vision Transformer 3D. We have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development.
  arXiv  Detail & Related papers  (2024-04-30T10:41:23Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• Leveraging Brain Modularity Prior for Interpretable Representation Learning of fMRI [38.236414924531196]
  Resting-state functional magnetic resonance imaging (rs-fMRI) can reflect spontaneous neural activities in brain. Previous studies propose to extract fMRI representations through diverse machine/deep learning methods for subsequent analysis. This paper proposes a Brain Modularity-constrained dynamic Representation learning (BMR) framework for interpretable fMRI analysis.
  arXiv  Detail & Related papers  (2023-06-24T23:45:47Z)
• Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities [31.448924808940284]
  We introduce a two-phase fMRI representation learning framework. The first phase pre-trains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder.
  arXiv  Detail & Related papers  (2023-05-26T19:16:23Z)
• BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding [51.911473457195555]
  BrainCLIP is a task-agnostic fMRI-based brain decoding model. It bridges the modality gap between brain activity, image, and text. BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
  arXiv  Detail & Related papers  (2023-02-25T03:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.