Brain Diffusion for Visual Exploration: Cortical Discovery using Large
Scale Generative Models
- URL: http://arxiv.org/abs/2306.03089v2
- Date: Tue, 28 Nov 2023 18:59:46 GMT
- Title: Brain Diffusion for Visual Exploration: Cortical Discovery using Large
Scale Generative Models
- Authors: Andrew F. Luo, Margaret M. Henderson, Leila Wehbe, Michael J. Tarr
- Abstract summary: We introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings.
Our approach builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis.
These results advance our understanding of the fine-grained functional organization of human visual cortex.
- Score: 6.866437017874623
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: A long standing goal in neuroscience has been to elucidate the functional
organization of the brain. Within higher visual cortex, functional accounts
have remained relatively coarse, focusing on regions of interest (ROIs) and
taking the form of selectivity for broad categories such as faces, places,
bodies, food, or words. Because the identification of such ROIs has typically
relied on manually assembled stimulus sets consisting of isolated objects in
non-ecological contexts, exploring functional organization without robust a
priori hypotheses has been challenging. To overcome these limitations, we
introduce a data-driven approach in which we synthesize images predicted to
activate a given brain region using paired natural images and fMRI recordings,
bypassing the need for category-specific stimuli. Our approach -- Brain
Diffusion for Visual Exploration ("BrainDiVE") -- builds on recent generative
methods by combining large-scale diffusion models with brain-guided image
synthesis. Validating our method, we demonstrate the ability to synthesize
preferred images with appropriate semantic specificity for well-characterized
category-selective ROIs. We then show that BrainDiVE can characterize
differences between ROIs selective for the same high-level category. Finally we
identify novel functional subdivisions within these ROIs, validated with
behavioral data. These results advance our understanding of the fine-grained
functional organization of human visual cortex, and provide well-specified
constraints for further examination of cortical organization using
hypothesis-driven methods.
Related papers
- Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation [56.34634121544929]
In this study, we first construct the brain-effective network via the dynamic causal model.
We then introduce an interpretable graph learning framework termed Spatio-Temporal Embedding ODE (STE-ODE)
This framework incorporates specifically designed directed node embedding layers, aiming at capturing the dynamic interplay between structural and effective networks.
arXiv Detail & Related papers (2024-05-21T20:37:07Z) - MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
Brain decoding aims to reconstruct stimuli from acquired brain signals.
Currently, brain decoding is confined to a per-subject-per-model paradigm.
We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
arXiv Detail & Related papers (2024-04-11T15:46:42Z) - BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity [6.285481522918523]
We introduce a data-driven method that generates natural language descriptions for images predicted to maximally activate individual voxels of interest.
We validate our method through fine-grained voxel-level captioning across higher-order visual regions.
To demonstrate how our method enables scientific discovery, we perform exploratory investigations on the distribution of "person" representations in the brain.
arXiv Detail & Related papers (2023-10-06T17:59:53Z) - BDEC:Brain Deep Embedded Clustering model [10.560936895047321]
We develop an assumption-free model called as BDEC, which leverages the robust data fitting capability of deep learning.
By comparing with nine commonly used brain parcellation methods, the BDEC model demonstrates significantly superior performance.
These results suggest that the BDEC parcellation captures the functional characteristics of the brain and holds promise for future voxel-wise brain network analysis.
arXiv Detail & Related papers (2023-09-12T02:42:11Z) - Patched Diffusion Models for Unsupervised Anomaly Detection in Brain MRI [55.78588835407174]
We propose a method that reformulates the generation task of diffusion models as a patch-based estimation of healthy brain anatomy.
We evaluate our approach on data of tumors and multiple sclerosis lesions and demonstrate a relative improvement of 25.1% compared to existing baselines.
arXiv Detail & Related papers (2023-03-07T09:40:22Z) - Semantic Brain Decoding: from fMRI to conceptually similar image
reconstruction of visual stimuli [0.29005223064604074]
We propose a novel approach to brain decoding that also relies on semantic and contextual similarity.
We employ an fMRI dataset of natural image vision and create a deep learning decoding pipeline inspired by the existence of both bottom-up and top-down processes in human vision.
We produce reconstructions of visual stimuli that match the original content very well on a semantic level, surpassing the state of the art in previous literature.
arXiv Detail & Related papers (2022-12-13T16:54:08Z) - Interpretable Fusion Analytics Framework for fMRI Connectivity:
Self-Attention Mechanism and Latent Space Item-Response Model [0.0]
We propose a novel analytical framework that interprets the classification result from deep learning processes.
The application of this proposed framework to the four types of cognitive impairment shows that our approach is valid for determining the significant ROI functions.
arXiv Detail & Related papers (2022-07-04T17:01:18Z) - Deep Representations for Time-varying Brain Datasets [4.129225533930966]
This paper builds an efficient graph neural network model that incorporates both region-mapped fMRI sequences and structural connectivities as inputs.
We find good representations of the latent brain dynamics through learning sample-level adaptive adjacency matrices.
These modules can be easily adapted to and are potentially useful for other applications outside the neuroscience domain.
arXiv Detail & Related papers (2022-05-23T21:57:31Z) - Brain Cortical Functional Gradients Predict Cortical Folding Patterns
via Attention Mesh Convolution [51.333918985340425]
We develop a novel attention mesh convolution model to predict cortical gyro-sulcal segmentation maps on individual brains.
Experiments show that the prediction performance via our model outperforms other state-of-the-art models.
arXiv Detail & Related papers (2022-05-21T14:08:53Z) - Functional2Structural: Cross-Modality Brain Networks Representation
Learning [55.24969686433101]
Graph mining on brain networks may facilitate the discovery of novel biomarkers for clinical phenotypes and neurodegenerative diseases.
We propose a novel graph learning framework, known as Deep Signed Brain Networks (DSBN), with a signed graph encoder.
We validate our framework on clinical phenotype and neurodegenerative disease prediction tasks using two independent, publicly available datasets.
arXiv Detail & Related papers (2022-05-06T03:45:36Z) - Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization.
We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise.
We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
arXiv Detail & Related papers (2020-06-11T17:29:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.