Visual Image Reconstruction from Brain Activity via Latent Representation

• URL: http://arxiv.org/abs/2505.08429v2
• Date: Thu, 19 Jun 2025 12:37:28 GMT
• Title: Visual Image Reconstruction from Brain Activity via Latent Representation
• Authors: Yukiyasu Kamitani, Misato Tanaka, Ken Shirakawa,
• Abstract summary: Review traces the field's evolution from early classification approaches to sophisticated reconstructions.<n>We discuss the need for diverse datasets and refined evaluation metrics aligned with human perceptual judgments.<n>Visual image reconstruction offers promising insights into neural coding and enables new psychological measurements of visual experiences.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual image reconstruction, the decoding of perceptual content from brain activity into images, has advanced significantly with the integration of deep neural networks (DNNs) and generative models. This review traces the field's evolution from early classification approaches to sophisticated reconstructions that capture detailed, subjective visual experiences, emphasizing the roles of hierarchical latent representations, compositional strategies, and modular architectures. Despite notable progress, challenges remain, such as achieving true zero-shot generalization for unseen images and accurately modeling the complex, subjective aspects of perception. We discuss the need for diverse datasets, refined evaluation metrics aligned with human perceptual judgments, and compositional representations that strengthen model robustness and generalizability. Ethical issues, including privacy, consent, and potential misuse, are underscored as critical considerations for responsible development. Visual image reconstruction offers promising insights into neural coding and enables new psychological measurements of visual experiences, with applications spanning clinical diagnostics and brain-machine interfaces.

Related papers

• Mitigating Low-Level Visual Hallucinations Requires Self-Awareness: Database, Model and Training Strategy [53.07517728420411]
  We introduce the first instruction database specifically focused on hallucinations in low-level vision tasks.<n>We propose the Self-Awareness Failure Elimination (SAFEQA) model to improve the perception and comprehension abilities of the model in low-level vision tasks.<n>We conduct comprehensive experiments on low-level vision tasks, with the results demonstrating that our proposed method significantly enhances self-awareness of the model in these tasks and reduces hallucinations.
  arXiv  Detail & Related papers  (2025-03-26T16:05:01Z)
• Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models [10.615012396285337]
  We develop algorithms to enhance our understanding of visual processes by incorporating whole-brain activation maps. We first compare our method with state-of-the-art approaches to decoding visual processing and show improved predictive semantic accuracy by 43%.
  arXiv  Detail & Related papers  (2024-11-11T16:51:17Z)
• When Does Perceptual Alignment Benefit Vision Representations? [76.32336818860965]
  We investigate how aligning vision model representations to human perceptual judgments impacts their usability. We find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Brain-Streams: fMRI-to-Image Reconstruction with Multi-modal Guidance [3.74142789780782]
  We show how modern LDMs incorporate multi-modal guidance for structurally and semantically plausible image generations. Brain-Streams maps fMRI signals from brain regions to appropriate embeddings. We validate the reconstruction ability of Brain-Streams both quantitatively and qualitatively on a real fMRI dataset.
  arXiv  Detail & Related papers  (2024-09-18T16:19:57Z)
• Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI [36.181302575642306]
  Reconstructing visual imagination presents a greater challenge, with potentially revolutionary applications. For the first time, we have compiled a substantial dataset (around 6h of scans) on visual imagery. We train a modified version of an fMRI-to-image model and demonstrate the feasibility of reconstructing images from two modes of imagination.
  arXiv  Detail & Related papers  (2024-04-08T12:46:39Z)
• Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity [60.983327742457995]
  Reconstructing the viewed images from human brain activity bridges human and computer vision through the Brain-Computer Interface. We devise Psychometry, an omnifit model for reconstructing images from functional Magnetic Resonance Imaging (fMRI) obtained from different subjects.
  arXiv  Detail & Related papers  (2024-03-29T07:16:34Z)
• MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion [7.597218661195779]
  We propose a two-stage image reconstruction model called MindDiffuser. In Stage 1, the VQ-VAE latent representations and the CLIP text embeddings decoded from fMRI are put into Stable Diffusion. In Stage 2, we utilize the CLIP visual feature decoded from fMRI as supervisory information, and continually adjust the two feature vectors decoded in Stage 1 through backpagation to align the structural information.
  arXiv  Detail & Related papers  (2023-08-08T13:28:34Z)
• Controllable Mind Visual Diffusion Model [58.83896307930354]
  Brain signal visualization has emerged as an active research area, serving as a critical interface between the human visual system and computer vision models. We propose a novel approach, referred to as Controllable Mind Visual Model Diffusion (CMVDM) CMVDM extracts semantic and silhouette information from fMRI data using attribute alignment and assistant networks. We then leverage a control model to fully exploit the extracted information for image synthesis, resulting in generated images that closely resemble the visual stimuli in terms of semantics and silhouette.
  arXiv  Detail & Related papers  (2023-05-17T11:36:40Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding [51.911473457195555]
  BrainCLIP is a task-agnostic fMRI-based brain decoding model. It bridges the modality gap between brain activity, image, and text. BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
  arXiv  Detail & Related papers  (2023-02-25T03:28:54Z)
• Exploring CLIP for Assessing the Look and Feel of Images [87.97623543523858]
  We introduce Contrastive Language-Image Pre-training (CLIP) models for assessing both the quality perception (look) and abstract perception (feel) of images in a zero-shot manner. Our results show that CLIP captures meaningful priors that generalize well to different perceptual assessments.
  arXiv  Detail & Related papers  (2022-07-25T17:58:16Z)
• Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN [16.169414324390218]
  Reconstructing seeing images from fMRI recordings is an absorbing research area in neuroscience. Visual encoding in brain is highly complex and not fully revealed. Inspired by the theory that visual features are hierarchically represented in cortex, we propose to break the complex visual signals into multi-level components.
  arXiv  Detail & Related papers  (2021-01-28T16:04:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.