Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain
Activities
- URL: http://arxiv.org/abs/2305.17214v4
- Date: Wed, 27 Dec 2023 09:39:41 GMT
- Title: Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain
Activities
- Authors: Jingyuan Sun, Mingxiao Li, Zijiao Chen, Yunhao Zhang, Shaonan Wang,
Marie-Francine Moens
- Abstract summary: We introduce a two-phase fMRI representation learning framework.
The first phase pre-trains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations.
The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder.
- Score: 31.448924808940284
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Decoding visual stimuli from neural responses recorded by functional Magnetic
Resonance Imaging (fMRI) presents an intriguing intersection between cognitive
neuroscience and machine learning, promising advancements in understanding
human visual perception and building non-invasive brain-machine interfaces.
However, the task is challenging due to the noisy nature of fMRI signals and
the intricate pattern of brain visual representations. To mitigate these
challenges, we introduce a two-phase fMRI representation learning framework.
The first phase pre-trains an fMRI feature learner with a proposed
Double-contrastive Mask Auto-encoder to learn denoised representations. The
second phase tunes the feature learner to attend to neural activation patterns
most informative for visual reconstruction with guidance from an image
auto-encoder. The optimized fMRI feature learner then conditions a latent
diffusion model to reconstruct image stimuli from brain activities.
Experimental results demonstrate our model's superiority in generating
high-resolution and semantically accurate images, substantially exceeding
previous state-of-the-art methods by 39.34% in the 50-way-top-1 semantic
classification accuracy. Our research invites further exploration of the
decoding task's potential and contributes to the development of non-invasive
brain-machine interfaces.
Related papers
- MindFormer: A Transformer Architecture for Multi-Subject Brain Decoding via fMRI [50.55024115943266]
We introduce a new Transformer architecture called MindFormer to generate fMRI-conditioned feature vectors.
MindFormer incorporates two key innovations: 1) a novel training strategy based on the IP-Adapter to extract semantically meaningful features from fMRI signals, and 2) a subject specific token and linear layer that effectively capture individual differences in fMRI signals.
arXiv Detail & Related papers (2024-05-28T00:36:25Z) - MindShot: Brain Decoding Framework Using Only One Image [21.53687547774089]
MindShot is proposed to achieve effective few-shot brain decoding by leveraging cross-subject prior knowledge.
New subjects and pretrained individuals only need to view images of the same semantic class, significantly expanding the model's applicability.
arXiv Detail & Related papers (2024-05-24T07:07:06Z) - Neural 3D decoding for human vision diagnosis [76.41771117405973]
We show how AI can go beyond the current state of the art by advancing from 2D visuals to visually plausible and functionally more comprehensive 3D visuals decoded from brain signals.
We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject who was presented with a 2D image, and yields as output the corresponding 3D object visuals.
arXiv Detail & Related papers (2024-05-24T06:06:11Z) - MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
Brain decoding aims to reconstruct stimuli from acquired brain signals.
Currently, brain decoding is confined to a per-subject-per-model paradigm.
We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
arXiv Detail & Related papers (2024-04-11T15:46:42Z) - fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for
Multi-Subject Brain Activity Decoding [54.17776744076334]
We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training.
Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns.
Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
arXiv Detail & Related papers (2023-11-01T07:24:22Z) - Decoding Realistic Images from Brain Activity with Contrastive
Self-supervision and Latent Diffusion [29.335943994256052]
Reconstructing visual stimuli from human brain activities provides a promising opportunity to advance our understanding of the brain's visual system.
We propose a two-phase framework named Contrast and Diffuse (CnD) to decode realistic images from functional magnetic resonance imaging (fMRI) recordings.
arXiv Detail & Related papers (2023-09-30T09:15:22Z) - MindDiffuser: Controlled Image Reconstruction from Human Brain Activity
with Semantic and Structural Diffusion [7.597218661195779]
We propose a two-stage image reconstruction model called MindDiffuser.
In Stage 1, the VQ-VAE latent representations and the CLIP text embeddings decoded from fMRI are put into Stable Diffusion.
In Stage 2, we utilize the CLIP visual feature decoded from fMRI as supervisory information, and continually adjust the two feature vectors decoded in Stage 1 through backpagation to align the structural information.
arXiv Detail & Related papers (2023-08-08T13:28:34Z) - Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
We introduce a unified framework that addresses both fMRI decoding and encoding.
Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
arXiv Detail & Related papers (2023-03-26T14:14:58Z) - BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP
for Generic Natural Visual Stimulus Decoding [51.911473457195555]
BrainCLIP is a task-agnostic fMRI-based brain decoding model.
It bridges the modality gap between brain activity, image, and text.
BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
arXiv Detail & Related papers (2023-02-25T03:28:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.