SwiFT: Swin 4D fMRI Transformer
- URL: http://arxiv.org/abs/2307.05916v2
- Date: Tue, 31 Oct 2023 04:54:00 GMT
- Title: SwiFT: Swin 4D fMRI Transformer
- Authors: Peter Yongho Kim, Junbeom Kwon, Sunghwan Joo, Sangyoon Bae, Donggyu
Lee, Yoonho Jung, Shinjae Yoo, Jiook Cha, Taesup Moon
- Abstract summary: We present SwiFTS (win 4D fMRI Transformer), a Swin Transformer architecture that can learn brain dynamics directly from volumes fMRI.
We evaluate SwiFT using multiple large-scale resting-state fMRI datasets to predict sex age and cognitive intelligence.
- Score: 17.95502427633986
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Modeling spatiotemporal brain dynamics from high-dimensional data, such as
functional Magnetic Resonance Imaging (fMRI), is a formidable task in
neuroscience. Existing approaches for fMRI analysis utilize hand-crafted
features, but the process of feature extraction risks losing essential
information in fMRI scans. To address this challenge, we present SwiFT (Swin 4D
fMRI Transformer), a Swin Transformer architecture that can learn brain
dynamics directly from fMRI volumes in a memory and computation-efficient
manner. SwiFT achieves this by implementing a 4D window multi-head
self-attention mechanism and absolute positional embeddings. We evaluate SwiFT
using multiple large-scale resting-state fMRI datasets, including the Human
Connectome Project (HCP), Adolescent Brain Cognitive Development (ABCD), and UK
Biobank (UKB) datasets, to predict sex, age, and cognitive intelligence. Our
experimental outcomes reveal that SwiFT consistently outperforms recent
state-of-the-art models. Furthermore, by leveraging its end-to-end learning
capability, we show that contrastive loss-based self-supervised pre-training of
SwiFT can enhance performance on downstream tasks. Additionally, we employ an
explainable AI method to identify the brain regions associated with sex
classification. To our knowledge, SwiFT is the first Swin Transformer
architecture to process dimensional spatiotemporal brain functional data in an
end-to-end fashion. Our work holds substantial potential in facilitating
scalable learning of functional brain imaging in neuroscience research by
reducing the hurdles associated with applying Transformer models to
high-dimensional fMRI.
Related papers
- Brain3D: Generating 3D Objects from fMRI [76.41771117405973]
We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject.
We show that our model captures the distinct functionalities of each region of human vision system.
Preliminary evaluations indicate that Brain3D can successfully identify the disordered brain regions in simulated scenarios.
arXiv Detail & Related papers (2024-05-24T06:06:11Z) - Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation [56.34634121544929]
In this study, we first construct the brain-effective network via the dynamic causal model.
We then introduce an interpretable graph learning framework termed Spatio-Temporal Embedding ODE (STE-ODE)
This framework incorporates specifically designed directed node embedding layers, aiming at capturing the dynamic interplay between structural and effective networks.
arXiv Detail & Related papers (2024-05-21T20:37:07Z) - ACTION: Augmentation and Computation Toolbox for Brain Network Analysis with Functional MRI [28.639321546348654]
Action is a Python-based and cross-platform toolbox for fMRI analysis.
It enables automatic fMRI augmentation, covering blood-oxygen-level-dependent (BOLD) signal augmentation and brain network augmentation.
It supports constructing deep learning models, which leverage large-scale auxiliary unlabeled data.
arXiv Detail & Related papers (2024-05-10T01:45:09Z) - MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
Brain decoding aims to reconstruct stimuli from acquired brain signals.
Currently, brain decoding is confined to a per-subject-per-model paradigm.
We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
arXiv Detail & Related papers (2024-04-11T15:46:42Z) - Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI [12.203617776046169]
We introduce a novel framework named Brainformer to analyze fMRI patterns in the human perception system.
This work introduces a prospective approach to transfer knowledge from human perception to neural networks.
arXiv Detail & Related papers (2023-11-30T22:39:23Z) - fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for
Multi-Subject Brain Activity Decoding [54.17776744076334]
We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training.
Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns.
Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
arXiv Detail & Related papers (2023-11-01T07:24:22Z) - BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP
for Generic Natural Visual Stimulus Decoding [51.911473457195555]
BrainCLIP is a task-agnostic fMRI-based brain decoding model.
It bridges the modality gap between brain activity, image, and text.
BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
arXiv Detail & Related papers (2023-02-25T03:28:54Z) - fMRI from EEG is only Deep Learning away: the use of interpretable DL to
unravel EEG-fMRI relationships [68.8204255655161]
We present an interpretable domain grounded solution to recover the activity of several subcortical regions from multichannel EEG data.
We recover individual spatial and time-frequency patterns of scalp EEG predictive of the hemodynamic signal in the subcortical nuclei.
arXiv Detail & Related papers (2022-10-23T15:11:37Z) - EEG to fMRI Synthesis: Is Deep Learning a candidate? [0.913755431537592]
This work provides the first comprehensive on how to use state-of-the-art principles from Neural Processing to synthesize fMRI data from electroencephalographic (EEG) view data.
A comparison of state-of-the-art synthesis approaches, including Autoencoders, Generative Adrial Networks and Pairwise Learning, is undertaken.
Results highlight the feasibility of EEG to fMRI brain image mappings, pinpointing the role of current advances in Machine Learning and showing the relevance of upcoming contributions to further improve performance.
arXiv Detail & Related papers (2020-09-29T16:29:20Z) - Attend and Decode: 4D fMRI Task State Decoding Using Attention Models [2.6954666679827137]
We present a novel architecture called Brain Attend and Decode (BAnD)
BAnD uses residual convolutional neural networks for spatial feature extraction and self-attention mechanisms temporal modeling.
We achieve significant performance gain compared to previous works on a 7-task benchmark from the Human Connectome Project-Young Adult dataset.
arXiv Detail & Related papers (2020-04-10T21:29:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.