Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals

• URL: http://arxiv.org/abs/2308.02510v2
• Date: Wed, 16 Aug 2023 09:59:40 GMT
• Title: Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals
• Authors: Yu-Ting Lan, Kan Ren, Yansen Wang, Wei-Long Zheng, Dongsheng Li, Bao-Liang Lu, Lili Qiu
• Abstract summary: We propose a comprehensive pipeline, named NeuroImagen, for reconstructing visual stimuli images from EEG signals. We incorporate a novel multi-level perceptual information decoding to draw multi-grained outputs from the given EEG data.
• Score: 27.92796103924193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Seeing is believing, however, the underlying mechanism of how human visual perceptions are intertwined with our cognitions is still a mystery. Thanks to the recent advances in both neuroscience and artificial intelligence, we have been able to record the visually evoked brain activities and mimic the visual perception ability through computational approaches. In this paper, we pay attention to visual stimuli reconstruction by reconstructing the observed images based on portably accessible brain signals, i.e., electroencephalography (EEG) data. Since EEG signals are dynamic in the time-series format and are notorious to be noisy, processing and extracting useful information requires more dedicated efforts; In this paper, we propose a comprehensive pipeline, named NeuroImagen, for reconstructing visual stimuli images from EEG signals. Specifically, we incorporate a novel multi-level perceptual information decoding to draw multi-grained outputs from the given EEG data. A latent diffusion model will then leverage the extracted information to reconstruct the high-resolution visual stimuli images. The experimental results have illustrated the effectiveness of image reconstruction and superior quantitative performance of our proposed method.

Related papers

• MindFormer: A Transformer Architecture for Multi-Subject Brain Decoding via fMRI [50.55024115943266]
  We introduce a new Transformer architecture called MindFormer to generate fMRI-conditioned feature vectors. MindFormer incorporates two key innovations: 1) a novel training strategy based on the IP-Adapter to extract semantically meaningful features from fMRI signals, and 2) a subject specific token and linear layer that effectively capture individual differences in fMRI signals.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• Neural 3D decoding for human vision diagnosis [76.41771117405973]
  We show how AI can go beyond the current state of the art by advancing from 2D visuals to visually plausible and functionally more comprehensive 3D visuals decoded from brain signals. We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject who was presented with a 2D image, and yields as output the corresponding 3D object visuals.
  arXiv  Detail & Related papers  (2024-05-24T06:06:11Z)
• Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI [36.181302575642306]
  Reconstructing visual imagination presents a greater challenge, with potentially revolutionary applications. For the first time, we have compiled a substantial dataset (around 6h of scans) on visual imagery. We train a modified version of an fMRI-to-image model and demonstrate the feasibility of reconstructing images from two modes of imagination.
  arXiv  Detail & Related papers  (2024-04-08T12:46:39Z)
• Reconstructing Visual Stimulus Images from EEG Signals Based on Deep Visual Representation Model [5.483279087074447]
  We propose a novel image reconstruction method based on EEG signals in this paper. To satisfy the high recognizability of visual stimulus images in fast switching manner, we build a visual stimuli image dataset. Deep visual representation model(DVRM) consisting of a primary encoder and a subordinate decoder is proposed to reconstruct visual stimuli.
  arXiv  Detail & Related papers  (2024-03-11T09:19:09Z)
• BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction [8.206564266319388]
  We propose a novel approach to reconstruct visual stimuli from EEG signals. We apply a self-supervised approach on the EEG signals to obtain EEG time-domain features. We also align EEG time-frequency embeddings with the coarse and fine-grained semantics in the CLIP space. Our proposed BrainVis outperforms state of the arts in both semantic fidelity reconstruction and generation quality.
  arXiv  Detail & Related papers  (2023-12-22T17:49:11Z)
• Learning Robust Deep Visual Representations from EEG Brain Recordings [13.768240137063428]
  This study proposes a two-stage method where the first step is to obtain EEG-derived features for robust learning of deep representations. We demonstrate the generalizability of our feature extraction pipeline across three different datasets using deep-learning architectures. We propose a novel framework to transform unseen images into the EEG space and reconstruct them with approximation.
  arXiv  Detail & Related papers  (2023-10-25T10:26:07Z)
• Brain Captioning: Decoding human brain activity into images and text [1.5486926490986461]
  We present an innovative method for decoding brain activity into meaningful images and captions. Our approach takes advantage of cutting-edge image captioning models and incorporates a unique image reconstruction pipeline. We evaluate our methods using quantitative metrics for both generated captions and images.
  arXiv  Detail & Related papers  (2023-05-19T09:57:19Z)
• Controllable Mind Visual Diffusion Model [58.83896307930354]
  Brain signal visualization has emerged as an active research area, serving as a critical interface between the human visual system and computer vision models. We propose a novel approach, referred to as Controllable Mind Visual Model Diffusion (CMVDM) CMVDM extracts semantic and silhouette information from fMRI data using attribute alignment and assistant networks. We then leverage a control model to fully exploit the extracted information for image synthesis, resulting in generated images that closely resemble the visual stimuli in terms of semantics and silhouette.
  arXiv  Detail & Related papers  (2023-05-17T11:36:40Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• BI AVAN: Brain inspired Adversarial Visual Attention Network [67.05560966998559]
  We propose a brain-inspired adversarial visual attention network (BI-AVAN) to characterize human visual attention directly from functional brain activity. Our model imitates the biased competition process between attention-related/neglected objects to identify and locate the visual objects in a movie frame the human brain focuses on in an unsupervised manner.
  arXiv  Detail & Related papers  (2022-10-27T22:20:36Z)
• Adapting Brain-Like Neural Networks for Modeling Cortical Visual Prostheses [68.96380145211093]
  Cortical prostheses are devices implanted in the visual cortex that attempt to restore lost vision by electrically stimulating neurons. Currently, the vision provided by these devices is limited, and accurately predicting the visual percepts resulting from stimulation is an open challenge. We propose to address this challenge by utilizing 'brain-like' convolutional neural networks (CNNs), which have emerged as promising models of the visual system.
  arXiv  Detail & Related papers  (2022-09-27T17:33:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.