Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation

• URL: http://arxiv.org/abs/2606.06345v1
• Date: Thu, 04 Jun 2026 16:18:08 GMT
• Title: Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation
• Abstract summary: We investigate whether and when brain decoding can be boosted by augmenting small fMRI datasets with synthetic data.<n>We use TRIBE v2, a large encoding model pretrained on more than 1000 hours of fMRI responses to video, audio and language.<n>Our results show up to 68% improvement in Top-10 image-retrieval accuracy compared to decoders trained only on real data.
• Score: 9.825779133835566
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Brain decoding is limited by the availability of labeled neural data, and remains challenging in low-data regimes. To address this issue, we investigate whether and when brain decoding can be boosted by augmenting small fMRI datasets with synthetic data generated by a pretrained model of fMRI responses to stimuli. We use TRIBE v2, a large encoding model pretrained on more than 1000 hours of fMRI responses to video, audio and language. For each dataset, we evaluate systematic grids that show how the performance of image decoders varies with the amount of synthetic data used for training. Our results, based on two datasets (the 7T fMRI Natural Scenes Dataset and 3T fMRI BOLD5000), show up to 68% improvement in Top-10 image-retrieval accuracy compared to decoders trained only on real data. Importantly, the proportion of augmented data required to reach a given image decoding performance needs to be adjusted depending on the data source. Surprisingly, image decoders trained exclusively on synthetic fMRI can perform above chance in some settings, suggesting that TRIBE v2 can support zero-shot brain-to-image decoding. Together, these results show how large-scale models of the fMRI responses to sight, sound and language may provide a foundation to improve the data efficiency for image decoding.

Related papers

• Adapting HFMCA to Graph Data: Self-Supervised Learning for Generalizable fMRI Representations [57.054499278843856]
  Functional magnetic resonance imaging (fMRI) analysis faces significant challenges due to limited dataset sizes and domain variability between studies.<n>Traditional self-supervised learning methods inspired by computer vision often rely on positive and negative sample pairs.<n>We propose adapting a recently developed Hierarchical Functional Maximal Correlation Algorithm (HFMCA) to graph-structured fMRI data.
  arXiv  Detail & Related papers  (2025-10-05T12:35:01Z)
• MRI Image Generation Based on Text Prompts [0.0]
  This study explores the use of text-prompted MRI image generation with the Stable Diffusion (SD) model to address challenges in acquiring real MRI datasets.<n>The SD model, pre-trained on natural images, was fine-tuned using the 3T fastMRI dataset and the 0.3T M4Raw dataset.<n>The performance of the fine-tuned model was evaluated using quantitative metrics, including Fr'echet Inception Distance (FID) and Multi-Scale Structural Similarity (MS-SSIM)
  arXiv  Detail & Related papers  (2025-05-23T03:01:22Z)
• PhaseGen: A Diffusion-Based Approach for Complex-Valued MRI Data Generation [1.683019219727036]
  Magnetic resonance imaging (MRI) raw data, or k-Space data, is complex-valued, containing both magnitude and phase information.<n>We introduce $textitPhaseGen$, a novel complex-valued diffusion model for generating synthetic MRI raw data conditioned on magnitude images.<n>Our results show that training with synthetic phase data significantly improves generalization for skull-stripping on real-world data.
  arXiv  Detail & Related papers  (2025-04-10T08:44:19Z)
• Scaling laws for decoding images from brain activity [7.864304771129752]
  Generative AI has recently propelled the decoding of images from brain activity.<n>How do these approaches scale with the amount and type of neural recordings?<n>Here, we systematically compare image decoding from four types of non-invasive devices.
  arXiv  Detail & Related papers  (2025-01-25T20:38:36Z)
• ContextMRI: Enhancing Compressed Sensing MRI through Metadata Conditioning [51.26601171361753]
  We propose ContextMRI, a text-conditioned diffusion model for MRI that integrates granular metadata into the reconstruction process.<n>We show that increasing the fidelity of metadata, ranging from slice location and contrast to patient age, sex, and pathology, systematically boosts reconstruction performance.
  arXiv  Detail & Related papers  (2025-01-08T05:15:43Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• Lite-Mind: Towards Efficient and Robust Brain Representation Network [23.132310312815967]
  Lite-Mind is a lightweight, efficient, and robust brain representation learning paradigm based on Discrete Frequency Transform (DFT) We show Lite-Mind achieves an impressive 94.6% fMRI-to-image retrieval accuracy on the NSD dataset for Subject 1, with 98.7% fewer parameters than MindEye. Lite-Mind is also proven to be able to be migrated to smaller fMRI datasets and establishes a new state-of-the-art for zero-shot classification on the GOD dataset.
  arXiv  Detail & Related papers  (2023-12-06T09:39:38Z)
• Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization [56.95046107046027]
  We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for scene coordinate regression. Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain.
  arXiv  Detail & Related papers  (2023-10-10T20:11:13Z)
• CMRxRecon: An open cardiac MRI dataset for the competition of accelerated image reconstruction [62.61209705638161]
  There has been growing interest in deep learning-based CMR imaging algorithms. Deep learning methods require large training datasets. This dataset includes multi-contrast, multi-view, multi-slice and multi-coil CMR imaging data from 300 subjects.
  arXiv  Detail & Related papers  (2023-09-19T15:14:42Z)
• Dataset Quantization [72.61936019738076]
  We present dataset quantization (DQ), a new framework to compress large-scale datasets into small subsets. DQ is the first method that can successfully distill large-scale datasets such as ImageNet-1k with a state-of-the-art compression ratio.
  arXiv  Detail & Related papers  (2023-08-21T07:24:29Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• Adversarial Distortion Learning for Medical Image Denoising [43.53912137735094]
  We present a novel adversarial distortion learning (ADL) for denoising two- and three-dimensional (2D/3D) biomedical image data. The proposed ADL consists of two auto-encoders: a denoiser and a discriminator. Both the denoiser and the discriminator are built upon a proposed auto-encoder called Efficient-Unet.
  arXiv  Detail & Related papers  (2022-04-29T13:47:39Z)
• Functional Magnetic Resonance Imaging data augmentation through conditional ICA [44.483210864902304]
  We introduce Conditional Independent Components Analysis (Conditional ICA): a fast functional Magnetic Resonance Imaging (fMRI) data augmentation technique. We show that Conditional ICA is successful at synthesizing data indistinguishable from observations, and that it yields gains in classification accuracy in brain decoding problems.
  arXiv  Detail & Related papers  (2021-07-11T22:36:14Z)
• Fed-Sim: Federated Simulation for Medical Imaging [131.56325440976207]
  We introduce a physics-driven generative approach that consists of two learnable neural modules. We show that our data synthesis framework improves the downstream segmentation performance on several datasets.
  arXiv  Detail & Related papers  (2020-09-01T19:17:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.