Reconstructing Visual Stimulus Images from EEG Signals Based on Deep Visual Representation Model

• URL: http://arxiv.org/abs/2403.06532v1
• Date: Mon, 11 Mar 2024 09:19:09 GMT
• Title: Reconstructing Visual Stimulus Images from EEG Signals Based on Deep Visual Representation Model
• Authors: Hongguang Pan, Zhuoyi Li, Yunpeng Fu, Xuebin Qin, Jianchen Hu
• Abstract summary: We propose a novel image reconstruction method based on EEG signals in this paper. To satisfy the high recognizability of visual stimulus images in fast switching manner, we build a visual stimuli image dataset. Deep visual representation model(DVRM) consisting of a primary encoder and a subordinate decoder is proposed to reconstruct visual stimuli.
• Score: 5.483279087074447
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing visual stimulus images is a significant task in neural decoding, and up to now, most studies consider the functional magnetic resonance imaging (fMRI) as the signal source. However, the fMRI-based image reconstruction methods are difficult to widely applied because of the complexity and high cost of the acquisition equipments. Considering the advantages of low cost and easy portability of the electroencephalogram (EEG) acquisition equipments, we propose a novel image reconstruction method based on EEG signals in this paper. Firstly, to satisfy the high recognizability of visual stimulus images in fast switching manner, we build a visual stimuli image dataset, and obtain the EEG dataset by a corresponding EEG signals collection experiment. Secondly, the deep visual representation model(DVRM) consisting of a primary encoder and a subordinate decoder is proposed to reconstruct visual stimuli. The encoder is designed based on the residual-in-residual dense blocks to learn the distribution characteristics between EEG signals and visual stimulus images, while the decoder is designed based on the deep neural network to reconstruct the visual stimulus image from the learned deep visual representation. The DVRM can fit the deep and multiview visual features of human natural state and make the reconstructed images more precise. Finally, we evaluate the DVRM in the quality of the generated images on our EEG dataset. The results show that the DVRM have good performance in the task of learning deep visual representation from EEG signals and generating reconstructed images that are realistic and highly resemble the original images.

Related papers

• MindFormer: A Transformer Architecture for Multi-Subject Brain Decoding via fMRI [50.55024115943266]
  We introduce a new Transformer architecture called MindFormer to generate fMRI-conditioned feature vectors. MindFormer incorporates two key innovations: 1) a novel training strategy based on the IP-Adapter to extract semantically meaningful features from fMRI signals, and 2) a subject specific token and linear layer that effectively capture individual differences in fMRI signals.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction [8.206564266319388]
  We propose a novel approach to reconstruct visual stimuli from EEG signals. We apply a self-supervised approach on the EEG signals to obtain EEG time-domain features. We also align EEG time-frequency embeddings with the coarse and fine-grained semantics in the CLIP space. Our proposed BrainVis outperforms state of the arts in both semantic fidelity reconstruction and generation quality.
  arXiv  Detail & Related papers  (2023-12-22T17:49:11Z)
• Learning Robust Deep Visual Representations from EEG Brain Recordings [13.768240137063428]
  This study proposes a two-stage method where the first step is to obtain EEG-derived features for robust learning of deep representations. We demonstrate the generalizability of our feature extraction pipeline across three different datasets using deep-learning architectures. We propose a novel framework to transform unseen images into the EEG space and reconstruct them with approximation.
  arXiv  Detail & Related papers  (2023-10-25T10:26:07Z)
• Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models [0.12289361708127873]
  We present an innovative method that employs to classify and reconstruct images from the ImageNet dataset using electroencephalography (EEG) data. We analyzed EEG recordings from 6 participants, each exposed to 50 images spanning 40 unique semantic categories. We incorporated an image reconstruction mechanism based on pre-trained latent diffusion models, which allowed us to generate an estimate of the images which had elicited EEG activity.
  arXiv  Detail & Related papers  (2023-09-08T09:13:50Z)
• Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals [27.92796103924193]
  We propose a comprehensive pipeline, named NeuroImagen, for reconstructing visual stimuli images from EEG signals. We incorporate a novel multi-level perceptual information decoding to draw multi-grained outputs from the given EEG data.
  arXiv  Detail & Related papers  (2023-07-27T12:54:16Z)
• Controllable Mind Visual Diffusion Model [58.83896307930354]
  Brain signal visualization has emerged as an active research area, serving as a critical interface between the human visual system and computer vision models. We propose a novel approach, referred to as Controllable Mind Visual Model Diffusion (CMVDM) CMVDM extracts semantic and silhouette information from fMRI data using attribute alignment and assistant networks. We then leverage a control model to fully exploit the extracted information for image synthesis, resulting in generated images that closely resemble the visual stimuli in terms of semantics and silhouette.
  arXiv  Detail & Related papers  (2023-05-17T11:36:40Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding [51.911473457195555]
  BrainCLIP is a task-agnostic fMRI-based brain decoding model. It bridges the modality gap between brain activity, image, and text. BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
  arXiv  Detail & Related papers  (2023-02-25T03:28:54Z)
• Attentive Symmetric Autoencoder for Brain MRI Segmentation [56.02577247523737]
  We propose a novel Attentive Symmetric Auto-encoder based on Vision Transformer (ViT) for 3D brain MRI segmentation tasks. In the pre-training stage, the proposed auto-encoder pays more attention to reconstruct the informative patches according to the gradient metrics. Experimental results show that our proposed attentive symmetric auto-encoder outperforms the state-of-the-art self-supervised learning methods and medical image segmentation models.
  arXiv  Detail & Related papers  (2022-09-19T09:43:19Z)
• Facial Image Reconstruction from Functional Magnetic Resonance Imaging via GAN Inversion with Improved Attribute Consistency [5.705640492618758]
  We propose a new framework to reconstruct facial images from fMRI data. The proposed framework accomplishes two goals: (1) reconstructing clear facial images from fMRI data and (2) maintaining the consistency of semantic characteristics.
  arXiv  Detail & Related papers  (2022-07-03T11:18:35Z)
• Adaptive Gradient Balancing for UndersampledMRI Reconstruction and Image-to-Image Translation [60.663499381212425]
  We enhance the image quality by using a Wasserstein Generative Adversarial Network combined with a novel Adaptive Gradient Balancing technique. In MRI, our method minimizes artifacts, while maintaining a high-quality reconstruction that produces sharper images than other techniques.
  arXiv  Detail & Related papers  (2021-04-05T13:05:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.