Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding

• URL: http://arxiv.org/abs/2507.22378v1
• Date: Wed, 30 Jul 2025 04:36:58 GMT
• Title: Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding
• Authors: Yueh-Po Peng, Vincent K. M. Cheung, Li Su,
• Abstract summary: We propose STDA-SwiFT, a transformer-based model that learns transferable representations from large-scale fMRI datasets.<n>We show that our model substantially improves downstream decoding performance of task-evoked activity.<n>Our work showcases transfer learning as a viable approach to overcome challenges in decoding brain activity from fMRI data.
• Score: 3.416130444086009
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A fundamental challenge in neuroscience is to decode mental states from brain activity. While functional magnetic resonance imaging (fMRI) offers a non-invasive approach to capture brain-wide neural dynamics with high spatial precision, decoding from fMRI data -- particularly from task-evoked activity -- remains challenging due to its high dimensionality, low signal-to-noise ratio, and limited within-subject data. Here, we leverage recent advances in computer vision and propose STDA-SwiFT, a transformer-based model that learns transferable representations from large-scale fMRI datasets via spatial-temporal divided attention and self-supervised contrastive learning. Using pretrained voxel-wise representations from 995 subjects in the Human Connectome Project (HCP), we show that our model substantially improves downstream decoding performance of task-evoked activity across multiple sensory and cognitive domains, even with minimal data preprocessing. We demonstrate performance gains from larger receptor fields afforded by our memory-efficient attention mechanism, as well as the impact of functional relevance in pretraining data when fine-tuning on small samples. Our work showcases transfer learning as a viable approach to harness large-scale datasets to overcome challenges in decoding brain activity from fMRI data.

Related papers

• Voxel-Level Brain States Prediction Using Swin Transformer [65.9194533414066]
  We propose a novel architecture which employs a 4D Shifted Window (Swin) Transformer as encoder to efficiently learn-temporal information and a convolutional decoder to enable brain state prediction at the same spatial and temporal resolution as the input fMRI data.<n>Our model has shown high accuracy when predicting 7.2s resting-state brain activities based on the prior 23.04s fMRI time series.<n>This shows promising evidence that thetemporal organization of the human brain can be learned by a Swin Transformer model, at high resolution, which provides a potential for reducing fMRI scan time and the development of brain-computer interfaces
  arXiv  Detail & Related papers  (2025-06-13T04:14:38Z)
• Generative forecasting of brain activity enhances Alzheimer's classification and interpretation [16.09844316281377]
  Resting-state functional magnetic resonance imaging (rs-fMRI) offers a non-invasive method to monitor neural activity. Deep learning has shown promise in capturing these representations. In this study, we focus on time series forecasting of independent component networks derived from rs-fMRI as a form of data augmentation.
  arXiv  Detail & Related papers  (2024-10-30T23:51:31Z)
• Scaling Wearable Foundation Models [54.93979158708164]
  We investigate the scaling properties of sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM. Our results establish the scaling laws of LSM for tasks such as imputation, extrapolation, both across time and sensor modalities.
  arXiv  Detail & Related papers  (2024-10-17T15:08:21Z)
• BrainMAE: A Region-aware Self-supervised Learning Framework for Brain Signals [11.030708270737964]
  We propose Brain Masked Auto-Encoder (BrainMAE) for learning representations directly from fMRI time-series data. BrainMAE consistently outperforms established baseline methods by significant margins in four distinct downstream tasks.
  arXiv  Detail & Related papers  (2024-06-24T19:16:24Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• Evaluating the structure of cognitive tasks with transfer learning [67.22168759751541]
  This study investigates the transferability of deep learning representations between different EEG decoding tasks. We conduct extensive experiments using state-of-the-art decoding models on two recently released EEG datasets.
  arXiv  Detail & Related papers  (2023-07-28T14:51:09Z)
• SwiFT: Swin 4D fMRI Transformer [17.95502427633986]
  We present SwiFTS (win 4D fMRI Transformer), a Swin Transformer architecture that can learn brain dynamics directly from volumes fMRI. We evaluate SwiFT using multiple large-scale resting-state fMRI datasets to predict sex age and cognitive intelligence.
  arXiv  Detail & Related papers  (2023-07-12T04:53:36Z)
• Deep Representations for Time-varying Brain Datasets [4.129225533930966]
  This paper builds an efficient graph neural network model that incorporates both region-mapped fMRI sequences and structural connectivities as inputs. We find good representations of the latent brain dynamics through learning sample-level adaptive adjacency matrices. These modules can be easily adapted to and are potentially useful for other applications outside the neuroscience domain.
  arXiv  Detail & Related papers  (2022-05-23T21:57:31Z)
• Learning Personal Representations from fMRIby Predicting Neurofeedback Performance [52.77024349608834]
  We present a deep neural network method for learning a personal representation for individuals performing a self neuromodulation task, guided by functional MRI (fMRI) The representation is learned by a self-supervised recurrent neural network, that predicts the Amygdala activity in the next fMRI frame given recent fMRI frames and is conditioned on the learned individual representation.
  arXiv  Detail & Related papers  (2021-12-06T10:16:54Z)
• Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations [60.47807856873544]
  A fundamental goal in neuroscience is to understand the relationship between neural activity and behavior. We generated a new multimodal dataset consisting of the spontaneous behaviors generated by fruit flies. This dataset and our new set of augmentations promise to accelerate the application of self-supervised learning methods in neuroscience.
  arXiv  Detail & Related papers  (2021-11-29T15:27:51Z)
• Functional Magnetic Resonance Imaging data augmentation through conditional ICA [44.483210864902304]
  We introduce Conditional Independent Components Analysis (Conditional ICA): a fast functional Magnetic Resonance Imaging (fMRI) data augmentation technique. We show that Conditional ICA is successful at synthesizing data indistinguishable from observations, and that it yields gains in classification accuracy in brain decoding problems.
  arXiv  Detail & Related papers  (2021-07-11T22:36:14Z)
• Attend and Decode: 4D fMRI Task State Decoding Using Attention Models [2.6954666679827137]
  We present a novel architecture called Brain Attend and Decode (BAnD) BAnD uses residual convolutional neural networks for spatial feature extraction and self-attention mechanisms temporal modeling. We achieve significant performance gain compared to previous works on a 7-task benchmark from the Human Connectome Project-Young Adult dataset.
  arXiv  Detail & Related papers  (2020-04-10T21:29:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.