Multiscale Voxel Based Decoding For Enhanced Natural Image Reconstruction From Brain Activity

• URL: http://arxiv.org/abs/2205.14177v1
• Date: Fri, 27 May 2022 18:09:07 GMT
• Title: Multiscale Voxel Based Decoding For Enhanced Natural Image Reconstruction From Brain Activity
• Authors: Mali Halac, Murat Isik, Hasan Ayaz, Anup Das
• Abstract summary: We present a novel approach for enhanced image reconstruction, in which existing methods for object decoding and image reconstruction are merged together. This is achieved by conditioning the reconstructed image to its decoded image category using a class-conditional generative adversarial network and neural style transfer. The results indicate that our approach improves the semantic similarity of the reconstructed images and can be used as a general framework for enhanced image reconstruction.
• Score: 0.22940141855172028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reconstructing perceived images from human brain activity monitored by functional magnetic resonance imaging (fMRI) is hard, especially for natural images. Existing methods often result in blurry and unintelligible reconstructions with low fidelity. In this study, we present a novel approach for enhanced image reconstruction, in which existing methods for object decoding and image reconstruction are merged together. This is achieved by conditioning the reconstructed image to its decoded image category using a class-conditional generative adversarial network and neural style transfer. The results indicate that our approach improves the semantic similarity of the reconstructed images and can be used as a general framework for enhanced image reconstruction.

Related papers

• Brain-optimized inference improves reconstructions of fMRI brain activity [0.0]
  We evaluate the prospect of further improving recent decoding methods by optimizing for consistency between reconstructions and brain activity during inference. We sample seed reconstructions from a base decoding method, then iteratively refine these reconstructions using a brain-optimized encoding model. We show that reconstruction quality can be significantly improved by explicitly aligning decoding to brain activity distributions.
  arXiv  Detail & Related papers  (2023-12-12T20:08:59Z)
• BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding [51.911473457195555]
  BrainCLIP is a task-agnostic fMRI-based brain decoding model. It bridges the modality gap between brain activity, image, and text. BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
  arXiv  Detail & Related papers  (2023-02-25T03:28:54Z)
• Facial Image Reconstruction from Functional Magnetic Resonance Imaging via GAN Inversion with Improved Attribute Consistency [5.705640492618758]
  We propose a new framework to reconstruct facial images from fMRI data. The proposed framework accomplishes two goals: (1) reconstructing clear facial images from fMRI data and (2) maintaining the consistency of semantic characteristics.
  arXiv  Detail & Related papers  (2022-07-03T11:18:35Z)
• Is Deep Image Prior in Need of a Good Education? [57.3399060347311]
  Deep image prior was introduced as an effective prior for image reconstruction. Despite its impressive reconstructive properties, the approach is slow when compared to learned or traditional reconstruction techniques. We develop a two-stage learning paradigm to address the computational challenge.
  arXiv  Detail & Related papers  (2021-11-23T15:08:26Z)
• Multi-modal Aggregation Network for Fast MR Imaging [85.25000133194762]
  We propose a novel Multi-modal Aggregation Network, named MANet, which is capable of discovering complementary representations from a fully sampled auxiliary modality. In our MANet, the representations from the fully sampled auxiliary and undersampled target modalities are learned independently through a specific network. Our MANet follows a hybrid domain learning framework, which allows it to simultaneously recover the frequency signal in the $k$-space domain.
  arXiv  Detail & Related papers  (2021-10-15T13:16:59Z)
• Using Deep Learning for Visual Decoding and Reconstruction from Brain Activity: A Review [0.0]
  I will show that these structures can struggle with adaptability to various input stimuli due to complicated objects in images. This paper will conclude the use of deep learning within visual decoding and reconstruction is highly optimal when using variations of deep neural networks.
  arXiv  Detail & Related papers  (2021-08-09T16:54:35Z)
• Adaptive Gradient Balancing for UndersampledMRI Reconstruction and Image-to-Image Translation [60.663499381212425]
  We enhance the image quality by using a Wasserstein Generative Adversarial Network combined with a novel Adaptive Gradient Balancing technique. In MRI, our method minimizes artifacts, while maintaining a high-quality reconstruction that produces sharper images than other techniques.
  arXiv  Detail & Related papers  (2021-04-05T13:05:22Z)
• NAS-DIP: Learning Deep Image Prior with Neural Architecture Search [65.79109790446257]
  Recent work has shown that the structure of deep convolutional neural networks can be used as a structured image prior. We propose to search for neural architectures that capture stronger image priors. We search for an improved network by leveraging an existing neural architecture search algorithm.
  arXiv  Detail & Related papers  (2020-08-26T17:59:36Z)
• Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation [181.08127307338654]
  This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. The deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images.
  arXiv  Detail & Related papers  (2020-03-30T17:45:07Z)
• BigGAN-based Bayesian reconstruction of natural images from human brain activity [14.038605815510145]
  We propose a new GAN-based visual reconstruction method (GAN-BVRM) that includes a classifier to decode categories from fMRI data. GAN-BVRM employs the pre-trained generator of the prevailing BigGAN to generate masses of natural images. Experimental results revealed that GAN-BVRM improves the fidelity and naturalness, that is, the reconstruction is natural and similar to the presented image stimuli.
  arXiv  Detail & Related papers  (2020-03-13T04:32:11Z)
• Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN [2.0754848504005583]
  We employ a recently proposed large-scale bi-directional adversarial network, called BigBiGAN, to decode and reconstruct natural scenes from fMRI patterns. We computed a linear mapping between fMRI data, acquired over images from 150 different categories of ImageNet, and their corresponding BigBiGAN latent vectors. We applied this mapping to the fMRI activity patterns obtained from 50 new test images from 50 unseen categories in order to retrieve their latent vectors, and reconstruct the corresponding images.
  arXiv  Detail & Related papers  (2020-01-31T10:46:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.