Real-time Reconstruction of Human Visual Perception from fMRI

• URL: http://arxiv.org/abs/2607.22753v1
• Date: Thu, 23 Jul 2026 14:54:28 GMT
• Title: Real-time Reconstruction of Human Visual Perception from fMRI
• Abstract summary: We present a real-time compatible adaptation of a computationally intensive state-of-the-art pipeline for reconstructing perceived natural images (MindEye2)<n>We performed a real-time scan where we decoded single-trial visual perception within seconds after an image was shown to the participant.<n>This work serves as a proof-of-concept that it is feasible to deploy these powerful fMRI decoding pipelines in real-time analysis.
• Score: 2.065551553722109
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time closed-loop neurofeedback based on functional magnetic resonance imaging (fMRI) has led to important scientific and clinical advances. However, the sophistication of the analysis methods used in real-time fMRI lags behind the state-of-the-art in fMRI decoding, largely due to computational factors: Most advanced decoding pipelines do not fit within the envelope of real-time processing, where the analysis needs to be conducted in a matter of seconds and without leveraging data acquired later in the session. Here, we present a real-time compatible adaptation of a computationally intensive state-of-the-art pipeline for reconstructing perceived natural images (MindEye2), and we demonstrate that reliable fine-grained decoding is still achievable in this setting. Using RT-Cloud, an open-source, scalable cloud-based platform, we performed a real-time scan where we decoded single-trial visual perception within seconds after an image was shown to the participant. Finally, we use simulated analyses to document the factors driving changes in performance from offline to real-time analysis. This work serves as a proof-of-concept that it is feasible to deploy these powerful fMRI decoding pipelines in real-time analysis, paving the way for their use in brain-computer interfaces for scientific discovery and clinical treatment.

Related papers

• Nonlocal operator learning for fMRI encoding and decoding tasks [1.8933894803085838]
  In this work, we investigate neural integral-operator models for encoding and decoding tasks in fMRI.<n>We evaluate our model on two open-source fMRI datasets.
  arXiv  Detail & Related papers  (2026-05-19T18:39:10Z)
• Accelerating 3D Photoacoustic Computed Tomography with End-to-End Physics-Aware Neural Operators [74.65171736966131]
  Photoacoustic computed tomography (PACT) combines optical contrast with ultrasonic resolution, achieving deep-tissue imaging beyond the optical diffusion limit.<n>Current implementations require dense transducer arrays and prolonged acquisition times, limiting clinical translation.<n>We introduce Pano, an end-to-end physics-aware model that directly learns the inverse acoustic mapping from sensor measurements to volumetric reconstructions.
  arXiv  Detail & Related papers  (2025-09-11T23:12:55Z)
• ContextMRI: Enhancing Compressed Sensing MRI through Metadata Conditioning [51.26601171361753]
  We propose ContextMRI, a text-conditioned diffusion model for MRI that integrates granular metadata into the reconstruction process.<n>We show that increasing the fidelity of metadata, ranging from slice location and contrast to patient age, sex, and pathology, systematically boosts reconstruction performance.
  arXiv  Detail & Related papers  (2025-01-08T05:15:43Z)
• TEAM PILOT -- Learned Feasible Extendable Set of Dynamic MRI Acquisition Trajectories [2.7719338074999547]
  We introduce a novel deep-compressed sensing approach that uses 3D window attention and flexible, temporally extendable acquisition trajectories. Our method significantly reduces both training and inference times compared to existing approaches. Tests with real data show that our approach outperforms current state-of-theart techniques.
  arXiv  Detail & Related papers  (2024-09-19T13:45:13Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• Implicit Neural Networks with Fourier-Feature Inputs for Free-breathing Cardiac MRI Reconstruction [21.261567937245808]
  We propose a reconstruction approach based on representing the beating heart with an implicit neural network and fitting the network so that the representation of the heart is consistent with the measurements. Our method achieves reconstruction quality on par with or slightly better than state-of-the-art untrained convolutional neural networks and superior image quality.
  arXiv  Detail & Related papers  (2023-05-11T14:14:30Z)
• A Path Towards Clinical Adaptation of Accelerated MRI [0.0]
  We explore augmentations to neural network MRI image reconstructors to enhance their clinical relevancy. We demonstrate that training reconstructors on MR signal data with variable acceleration factors can improve their average performance during a clinical patient scan by up to $2%$.
  arXiv  Detail & Related papers  (2022-08-26T18:34:41Z)
• A Long Short-term Memory Based Recurrent Neural Network for Interventional MRI Reconstruction [50.1787181309337]
  We propose a convolutional long short-term memory (Conv-LSTM) based recurrent neural network (RNN), or ConvLR, to reconstruct interventional images with golden-angle radial sampling. The proposed algorithm has the potential to achieve real-time i-MRI for DBS and can be used for general purpose MR-guided intervention.
  arXiv  Detail & Related papers  (2022-03-28T14:03:45Z)
• A parameter refinement method for Ptychography based on Deep Learning concepts [55.41644538483948]
  coarse parametrisation in propagation distance, position errors and partial coherence frequently menaces the experiment viability. A modern Deep Learning framework is used to correct autonomously the setup incoherences, thus improving the quality of a ptychography reconstruction. We tested our system on both synthetic datasets and also on real data acquired at the TwinMic beamline of the Elettra synchrotron facility.
  arXiv  Detail & Related papers  (2021-05-18T10:15:17Z)
• A DICOM Framework for Machine Learning Pipelines against Real-Time Radiology Images [50.222197963803644]
  Niffler is an integrated framework that enables the execution of machine learning pipelines at research clusters. Niffler uses the Digital Imaging and Communications in Medicine (DICOM) protocol to fetch and store imaging data. We present its architecture and three of its use cases: an inferior vena cava filter detection from the images in real-time, identification of scanner utilization, and scanner clock calibration.
  arXiv  Detail & Related papers  (2020-04-16T21:06:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.