Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding

• URL: http://arxiv.org/abs/2510.16870v1
• Date: Sun, 19 Oct 2025 15:11:03 GMT
• Title: Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding
• Authors: Yudan Ren, Xinlong Wang, Kexin Wang, Tian Xia, Zihan Ma, Zhaowei Li, Xiangrong Bi, Xiao Li, Xiaowei He,
• Abstract summary: Current understanding of the parallels between artificial neural networks (ANNs) and human brain processing remains limited.<n>We propose a novel neuron-level analysis framework that investigates the multimodal information processing mechanisms in vision-language models (VLMs) through the lens of human brain activity.
• Score: 34.313883741642066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While brain-inspired artificial intelligence(AI) has demonstrated promising results, current understanding of the parallels between artificial neural networks (ANNs) and human brain processing remains limited: (1) unimodal ANN studies fail to capture the brain's inherent multimodal processing capabilities, and (2) multimodal ANN research primarily focuses on high-level model outputs, neglecting the crucial role of individual neurons. To address these limitations, we propose a novel neuron-level analysis framework that investigates the multimodal information processing mechanisms in vision-language models (VLMs) through the lens of human brain activity. Our approach uniquely combines fine-grained artificial neuron (AN) analysis with fMRI-based voxel encoding to examine two architecturally distinct VLMs: CLIP and METER. Our analysis reveals four key findings: (1) ANs successfully predict biological neurons (BNs) activities across multiple functional networks (including language, vision, attention, and default mode), demonstrating shared representational mechanisms; (2) Both ANs and BNs demonstrate functional redundancy through overlapping neural representations, mirroring the brain's fault-tolerant and collaborative information processing mechanisms; (3) ANs exhibit polarity patterns that parallel the BNs, with oppositely activated BNs showing mirrored activation trends across VLM layers, reflecting the complexity and bidirectional nature of neural information processing; (4) The architectures of CLIP and METER drive distinct BNs: CLIP's independent branches show modality-specific specialization, whereas METER's cross-modal design yields unified cross-modal activation, highlighting the architecture's influence on ANN brain-like properties. These results provide compelling evidence for brain-like hierarchical processing in VLMs at the neuronal level.

Related papers

• Achieving Fine-grained Cross-modal Understanding through Brain-inspired Hierarchical Representation Learning [20.602643947067406]
  We present NeuroAlign, a novel framework for fine-grained fMRI-video alignment inspired by the hierarchical organization of the human visual system.<n>Our framework implements a two-stage mechanism that mirrors biological visual pathways: global semantic understanding and fine-grained pattern matching.<n> Experiments demonstrate that NeuroAlign significantly outperforms existing methods in cross-modal retrieval tasks.
  arXiv  Detail & Related papers  (2026-01-04T02:54:29Z)
• Application of an attention-based CNN-BiLSTM framework for in vivo two-photon calcium imaging of neuronal ensembles: decoding complex bilateral forelimb movements from unilateral M1 [0.511850618931844]
  Decoding, such as movement, from multiscale brain networks remains a central objective in neuroscience.<n>In this study, we employ a hybrid deep learning framework, an attention-based CNN-BiLSTM model, to decode skilled and complex forelimb movements.<n>Our findings demonstrate that the intricate movements of both ipsilateral and contralateral forelimbs can be accurately decoded from unilateral M1 neuronal ensembles.
  arXiv  Detail & Related papers  (2025-04-23T17:43:00Z)
• BrainMAP: Learning Multiple Activation Pathways in Brain Networks [77.15180533984947]
  We introduce a novel framework BrainMAP to learn Multiple Activation Pathways in Brain networks.<n>Our framework enables explanatory analyses of crucial brain regions involved in tasks.
  arXiv  Detail & Related papers  (2024-12-23T09:13:35Z)
• Brain-like Functional Organization within Large Language Models [58.93629121400745]
  The human brain has long inspired the pursuit of artificial intelligence (AI) Recent neuroimaging studies provide compelling evidence of alignment between the computational representation of artificial neural networks (ANNs) and the neural responses of the human brain to stimuli. In this study, we bridge this gap by directly coupling sub-groups of artificial neurons with functional brain networks (FBNs) This framework links the AN sub-groups to FBNs, enabling the delineation of brain-like functional organization within large language models (LLMs)
  arXiv  Detail & Related papers  (2024-10-25T13:15:17Z)
• Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling [52.06722364186432]
  We propose a biologically-informed framework for enhancing artificial neural networks (ANNs) Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors. We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
  arXiv  Detail & Related papers  (2024-07-05T14:11:28Z)
• BrainMAE: A Region-aware Self-supervised Learning Framework for Brain Signals [11.030708270737964]
  We propose Brain Masked Auto-Encoder (BrainMAE) for learning representations directly from fMRI time-series data. BrainMAE consistently outperforms established baseline methods by significant margins in four distinct downstream tasks.
  arXiv  Detail & Related papers  (2024-06-24T19:16:24Z)
• Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation [56.34634121544929]
  In this study, we first construct the brain-effective network via the dynamic causal model. We then introduce an interpretable graph learning framework termed Spatio-Temporal Embedding ODE (STE-ODE) This framework incorporates specifically designed directed node embedding layers, aiming at capturing the dynamic interplay between structural and effective networks.
  arXiv  Detail & Related papers  (2024-05-21T20:37:07Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• Multimodal foundation models are better simulators of the human brain [65.10501322822881]
  We present a newly-designed multimodal foundation model pre-trained on 15 million image-text pairs. We find that both visual and lingual encoders trained multimodally are more brain-like compared with unimodal ones.
  arXiv  Detail & Related papers  (2022-08-17T12:36:26Z)
• Functional2Structural: Cross-Modality Brain Networks Representation Learning [55.24969686433101]
  Graph mining on brain networks may facilitate the discovery of novel biomarkers for clinical phenotypes and neurodegenerative diseases. We propose a novel graph learning framework, known as Deep Signed Brain Networks (DSBN), with a signed graph encoder. We validate our framework on clinical phenotype and neurodegenerative disease prediction tasks using two independent, publicly available datasets.
  arXiv  Detail & Related papers  (2022-05-06T03:45:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.