Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG

• URL: http://arxiv.org/abs/2605.19224v1
• Date: Tue, 19 May 2026 00:49:51 GMT
• Title: Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG
• Abstract summary: We propose using non-invasive fMRI to bridge the gap in training data.<n>Using spoken language representations fine-tuned on fMRI, we build encoding models of ECoG.<n> Prediction improved in frequency bands well beyond what is directly measured in fMRI.
• Score: 45.06821993085712
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Neuroscientists have recently turned to intracranial brain recording methods, like electrocorticography (ECoG), for human experiments because of the fine spatial and temporal resolution that they afford. Models trained on this data, however, are fundamentally restricted by the patient populations that can receive the implants necessary for recording. We propose using non-invasive fMRI to bridge the gap in training data. Using spoken language representations fine-tuned on fMRI, we build encoding models of ECoG. These representations showed improved prediction performance in ECoG, even though the temporal resolution of fMRI is two orders of magnitude worse. Prediction improved in frequency bands well beyond what is directly measured in fMRI. Next, to test the procedure's generalization ability, we fine-tuned models on fMRI responses that were temporally downsampled by a factor of 2. Despite the loss in resolution, these models were able to predict fMRI and ECoG responses at levels comparable to the original fMRI-tuned models. Finally, we showed that ECoG performance steadily scales with the amount of fMRI-tuning data. Our results show that "slow" data like fMRI can be a valuable resource for building better models of "fast" brain data like ECoG. In the future, integrating across multiple recording methods may further improve performance in other applications, like decoding.

Related papers

• Estimating Brain Activity with High Spatial and Temporal Resolution using a Naturalistic MEG-fMRI Encoding Model [6.157087562708548]
  Current non-invasive neuroimaging techniques trade off between spatial resolution and temporal resolution.<n>We develop a transformer-based encoding model that combines the magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI)<n>Our model is trained to predict MEG and fMRI from multiple subjects, simultaneously with a latent layer that represents our estimates of reconstructed cortical sources.
  arXiv  Detail & Related papers  (2025-10-10T14:18:10Z)
• Adapting HFMCA to Graph Data: Self-Supervised Learning for Generalizable fMRI Representations [57.054499278843856]
  Functional magnetic resonance imaging (fMRI) analysis faces significant challenges due to limited dataset sizes and domain variability between studies.<n>Traditional self-supervised learning methods inspired by computer vision often rely on positive and negative sample pairs.<n>We propose adapting a recently developed Hierarchical Functional Maximal Correlation Algorithm (HFMCA) to graph-structured fMRI data.
  arXiv  Detail & Related papers  (2025-10-05T12:35:01Z)
• Predicting Cognition from fMRI:A Comparative Study of Graph, Transformer, and Kernel Models Across Task and Rest Conditions [1.0832932170181544]
  This study systematically benchmarked classical machine learning (KRR) and advanced deep learning (DL) models for cognitive prediction.<n>Our results revealed that task-based fMRI, eliciting neural responses directly tied to cognition, outperformed RS fMRI in predicting cognitive behavior.
  arXiv  Detail & Related papers  (2025-07-28T17:29:22Z)
• Voxel-Level Brain States Prediction Using Swin Transformer [65.9194533414066]
  We propose a novel architecture which employs a 4D Shifted Window (Swin) Transformer as encoder to efficiently learn-temporal information and a convolutional decoder to enable brain state prediction at the same spatial and temporal resolution as the input fMRI data.<n>Our model has shown high accuracy when predicting 7.2s resting-state brain activities based on the prior 23.04s fMRI time series.<n>This shows promising evidence that thetemporal organization of the human brain can be learned by a Swin Transformer model, at high resolution, which provides a potential for reducing fMRI scan time and the development of brain-computer interfaces
  arXiv  Detail & Related papers  (2025-06-13T04:14:38Z)
• Diffusion Probabilistic Generative Models for Accelerated, in-NICU Permanent Magnet Neonatal MRI [2.1400895396229913]
  Permanent magnet scanners operating in the neonatal intensive care unit (NICU) facilitate MRI of sick infants.<n>These scanners have long scan times due to lower signal-to-noise ratios (SNR) and limited receive coils.<n>This work accelerates in-NICU MRI with diffusion probabilistic generative models by developing a training pipeline.
  arXiv  Detail & Related papers  (2025-05-21T20:05:45Z)
• ContextMRI: Enhancing Compressed Sensing MRI through Metadata Conditioning [51.26601171361753]
  We propose ContextMRI, a text-conditioned diffusion model for MRI that integrates granular metadata into the reconstruction process.<n>We show that increasing the fidelity of metadata, ranging from slice location and contrast to patient age, sex, and pathology, systematically boosts reconstruction performance.
  arXiv  Detail & Related papers  (2025-01-08T05:15:43Z)
• A Generative Diffusion Model to Solve Inverse Problems for Robust in-NICU Neonatal MRI [2.508200203858861]
  We present the first acquisition-agnostic diffusion generative model for Magnetic Resonance Imaging (MRI) in the neonatal intensive care unit (NICU)<n>In-NICU MRI scanners leverage permanent magnets at lower field-strengths for non-invasive assessment of potential brain abnormalities.<n>In this setting, training data sizes are small and intrinsically suffer from low signal-to-noise ratio (SNR)
  arXiv  Detail & Related papers  (2024-10-28T23:12:09Z)
• A Unified Model for Compressed Sensing MRI Across Undersampling Patterns [69.19631302047569]
  We propose a unified MRI reconstruction model robust to various measurement undersampling patterns and image resolutions.<n>Our model improves SSIM by 11% and PSNR by 4 dB over a state-of-the-art CNN (End-to-End VarNet) with 600$times$ faster inference than diffusion methods.
  arXiv  Detail & Related papers  (2024-10-05T20:03:57Z)
• Spatiotemporal Graph Neural Network Modelling Perfusion MRI [12.712005118761516]
  Per vascular MRI (pMRI) offers valuable insights into tumority and promises to predict tumor genotypes. Yet effective models tailored to 4D pMRI are still lacking. This study presents the first attempt to model 4D pMRI using a GNN-based model.
  arXiv  Detail & Related papers  (2024-06-10T16:24:46Z)
• Generative AI for Rapid Diffusion MRI with Improved Image Quality, Reliability and Generalizability [3.6119644566822484]
  We employ a Swin UNEt Transformers model, trained on augmented Human Connectome Project data, to perform generalized denoising of dMRI. We demonstrate super-resolution with artificially downsampled HCP data in normal adult volunteers. We exceed current state-of-the-art denoising methods in accuracy and test-retest reliability of rapid diffusion tensor imaging requiring only 90 seconds of scan time.
  arXiv  Detail & Related papers  (2023-03-10T03:39:23Z)
• Data and Physics Driven Learning Models for Fast MRI -- Fundamentals and Methodologies from CNN, GAN to Attention and Transformers [72.047680167969]
  This article aims to introduce the deep learning based data driven techniques for fast MRI including convolutional neural network and generative adversarial network based methods. We will detail the research in coupling physics and data driven models for MRI acceleration. Finally, we will demonstrate through a few clinical applications, explain the importance of data harmonisation and explainable models for such fast MRI techniques in multicentre and multi-scanner studies.
  arXiv  Detail & Related papers  (2022-04-01T22:48:08Z)
• Robust Compressed Sensing MRI with Deep Generative Priors [84.69062247243953]
  We present the first successful application of the CSGM framework on clinical MRI data. We train a generative prior on brain scans from the fastMRI dataset, and show that posterior sampling via Langevin dynamics achieves high quality reconstructions.
  arXiv  Detail & Related papers  (2021-08-03T08:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.