Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models

• URL: http://arxiv.org/abs/2306.03089v2
• Date: Tue, 28 Nov 2023 18:59:46 GMT
• Title: Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models
• Authors: Andrew F. Luo, Margaret M. Henderson, Leila Wehbe, Michael J. Tarr
• Abstract summary: We introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings. Our approach builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. These results advance our understanding of the fine-grained functional organization of human visual cortex.
• Score: 6.866437017874623
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: A long standing goal in neuroscience has been to elucidate the functional organization of the brain. Within higher visual cortex, functional accounts have remained relatively coarse, focusing on regions of interest (ROIs) and taking the form of selectivity for broad categories such as faces, places, bodies, food, or words. Because the identification of such ROIs has typically relied on manually assembled stimulus sets consisting of isolated objects in non-ecological contexts, exploring functional organization without robust a priori hypotheses has been challenging. To overcome these limitations, we introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings, bypassing the need for category-specific stimuli. Our approach -- Brain Diffusion for Visual Exploration ("BrainDiVE") -- builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. Validating our method, we demonstrate the ability to synthesize preferred images with appropriate semantic specificity for well-characterized category-selective ROIs. We then show that BrainDiVE can characterize differences between ROIs selective for the same high-level category. Finally we identify novel functional subdivisions within these ROIs, validated with behavioral data. These results advance our understanding of the fine-grained functional organization of human visual cortex, and provide well-specified constraints for further examination of cortical organization using hypothesis-driven methods.

Related papers

• Brain-like Functional Organization within Large Language Models [58.93629121400745]
  The human brain has long inspired the pursuit of artificial intelligence (AI) Recent neuroimaging studies provide compelling evidence of alignment between the computational representation of artificial neural networks (ANNs) and the neural responses of the human brain to stimuli. In this study, we bridge this gap by directly coupling sub-groups of artificial neurons with functional brain networks (FBNs) This framework links the AN sub-groups to FBNs, enabling the delineation of brain-like functional organization within large language models (LLMs)
  arXiv  Detail & Related papers  (2024-10-25T13:15:17Z)
• Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers [5.265058307999745]
  We introduce BrainSAIL, a method for isolating neurally-activating visual concepts in images. BrainSAIL exploits semantically consistent, dense spatial features from pre-trained vision models. We validate BrainSAIL on cortical regions with known category selectivity.
  arXiv  Detail & Related papers  (2024-10-07T17:59:45Z)
• Knowledge-Guided Prompt Learning for Lifespan Brain MR Image Segmentation [53.70131202548981]
  We present a two-step segmentation framework employing Knowledge-Guided Prompt Learning (KGPL) for brain MRI. Specifically, we first pre-train segmentation models on large-scale datasets with sub-optimal labels. The introduction of knowledge-wise prompts captures semantic relationships between anatomical variability and biological processes.
  arXiv  Detail & Related papers  (2024-07-31T04:32:43Z)
• Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation [56.34634121544929]
  In this study, we first construct the brain-effective network via the dynamic causal model. We then introduce an interpretable graph learning framework termed Spatio-Temporal Embedding ODE (STE-ODE) This framework incorporates specifically designed directed node embedding layers, aiming at capturing the dynamic interplay between structural and effective networks.
  arXiv  Detail & Related papers  (2024-05-21T20:37:07Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity [6.285481522918523]
  We introduce a data-driven method that generates natural language descriptions for images predicted to maximally activate individual voxels of interest. We validate our method through fine-grained voxel-level captioning across higher-order visual regions. To demonstrate how our method enables scientific discovery, we perform exploratory investigations on the distribution of "person" representations in the brain.
  arXiv  Detail & Related papers  (2023-10-06T17:59:53Z)
• Interpretable Fusion Analytics Framework for fMRI Connectivity: Self-Attention Mechanism and Latent Space Item-Response Model [0.4893345190925178]
  We propose a novel analytical framework that interprets the classification result from deep learning processes. The application of this proposed framework to the four types of cognitive impairment shows that our approach is valid for determining the significant ROI functions.
  arXiv  Detail & Related papers  (2022-07-04T17:01:18Z)
• Deep Representations for Time-varying Brain Datasets [4.129225533930966]
  This paper builds an efficient graph neural network model that incorporates both region-mapped fMRI sequences and structural connectivities as inputs. We find good representations of the latent brain dynamics through learning sample-level adaptive adjacency matrices. These modules can be easily adapted to and are potentially useful for other applications outside the neuroscience domain.
  arXiv  Detail & Related papers  (2022-05-23T21:57:31Z)
• Brain Cortical Functional Gradients Predict Cortical Folding Patterns via Attention Mesh Convolution [51.333918985340425]
  We develop a novel attention mesh convolution model to predict cortical gyro-sulcal segmentation maps on individual brains. Experiments show that the prediction performance via our model outperforms other state-of-the-art models.
  arXiv  Detail & Related papers  (2022-05-21T14:08:53Z)
• Functional2Structural: Cross-Modality Brain Networks Representation Learning [55.24969686433101]
  Graph mining on brain networks may facilitate the discovery of novel biomarkers for clinical phenotypes and neurodegenerative diseases. We propose a novel graph learning framework, known as Deep Signed Brain Networks (DSBN), with a signed graph encoder. We validate our framework on clinical phenotype and neurodegenerative disease prediction tasks using two independent, publicly available datasets.
  arXiv  Detail & Related papers  (2022-05-06T03:45:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.