Interpretable EEG-to-Image Generation with Semantic Prompts

• URL: http://arxiv.org/abs/2507.07157v1
• Date: Wed, 09 Jul 2025 17:18:06 GMT
• Title: Interpretable EEG-to-Image Generation with Semantic Prompts
• Abstract summary: Our model bypasses direct EEG-to-image generation by aligning EEG signals with semantic captions.<n>A transformer-based EEG encoder maps brain activity to these captions through contrastive learning.<n>This text-mediated framework yields state-of-the-art visual decoding on the EEGCVPR dataset.
• Score: 6.712646807032639
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decoding visual experience from brain signals offers exciting possibilities for neuroscience and interpretable AI. While EEG is accessible and temporally precise, its limitations in spatial detail hinder image reconstruction. Our model bypasses direct EEG-to-image generation by aligning EEG signals with multilevel semantic captions -- ranging from object-level to abstract themes -- generated by a large language model. A transformer-based EEG encoder maps brain activity to these captions through contrastive learning. During inference, caption embeddings retrieved via projection heads condition a pretrained latent diffusion model for image generation. This text-mediated framework yields state-of-the-art visual decoding on the EEGCVPR dataset, with interpretable alignment to known neurocognitive pathways. Dominant EEG-caption associations reflected the importance of different semantic levels extracted from perceived images. Saliency maps and t-SNE projections reveal semantic topography across the scalp. Our model demonstrates how structured semantic mediation enables cognitively aligned visual decoding from EEG.

Related papers

• Neuroscience-inspired Staged Representation Learning with Disentangled Coarse- and Fine-Grained Semantics for EEG Visual Decoding [20.814486415153404]
  Existing EEG visual decoding methods mainly focus on learning a single global EEG embedding for cross-modal alignment.<n>We propose a neuroscience-inspired staged representation learning framework that reformulates EEG visual decoding as a stage-specific representation decomposition problem.<n>The proposed framework organizes EEG representation learning into three complementary phases: low-level visual representation learning, high-level semantic representation learning, and integrative information fusion.
  arXiv  Detail & Related papers  (2026-05-16T10:30:33Z)
• Structure-Guided Diffusion Model for EEG-Based Visual Cognition Reconstruction [2.77392799295445]
  Decoding visual information from electroencephalography (EEG) is an important problem in neuroscience and brain-computer interface (BCI) research.<n>We propose a Structure-Guided Diffusion Model (SGDM) that incorporates explicit structural information for EEG-based visual reconstruction.
  arXiv  Detail & Related papers  (2026-04-24T15:20:59Z)
• Spatial-Functional awareness Transformer-based graph archetype contrastive learning for Decoding Visual Neural Representations from EEG [3.661246946935037]
  We propose a Spatial-Functional Awareness Transformer-based Graph Archetype Contrastive Learning (SFTG) framework to enhance EEG-based visual decoding.<n>Specifically, we introduce the EEG Graph Transformer (EGT), a novel graph-based neural architecture that simultaneously encodes spatial brain connectivity and temporal neural dynamics.<n>To mitigate high intra-subject variability, we propose Graph Archetype Contrastive Learning (GAC), which learns subject-specific EEG graph archetypes to improve feature consistency and class separability.
  arXiv  Detail & Related papers  (2025-09-29T13:27:55Z)
• WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities [55.00677513249723]
  EEG signals simultaneously encode both cognitive processes and intrinsic neural states.<n>We map EEG signals and their corresponding modalities into a unified semantic space to achieve generalized interpretation.<n>The resulting model demonstrates robust classification accuracy while supporting flexible, open-ended conversations.
  arXiv  Detail & Related papers  (2025-09-26T06:21:51Z)
• Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models [6.761875482596085]
  We present the first image-to-brain signal framework that generates M/EEG from images.<n>The proposed framework comprises two key components: a pretrained CLIP visual encoder and a cross-attention enhanced U-Net diffusion model.<n>Unlike conventional generative models that rely on simple concatenation for conditioning, our cross-attention modules capture the complex interplay between visual features and brain signal representations.
  arXiv  Detail & Related papers  (2025-08-31T10:29:58Z)
• Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation [52.51005875755718]
  We focus on EEG-to-text decoding and address its hallucination issue through the lens of posterior collapse.<n>Acknowledging the underlying mismatch in information capacity between EEG and text, we reframe the decoding task as semantic summarization of core meanings.<n>Experiments on the public ZuCo dataset demonstrate that GLIM consistently generates fluent, EEG-grounded sentences.
  arXiv  Detail & Related papers  (2025-05-21T05:29:55Z)
• ViEEG: Hierarchical Visual Neural Representation for EEG Brain Decoding [18.51835182602402]
  ViEEG is a neuro-We further adopt hierarchical contrastive learning for EEG-CLIP representation alignment, enabling zero-shot object recognition.<n>Our framework sets a new paradigm for EEG brain decoding.<n>ViEEG decomposes each visual stimulus into three biologically aligned components-contour, foreground object, and contextual scene-serving anchors for a three-stream EEG encoder.
  arXiv  Detail & Related papers  (2025-05-18T13:19:08Z)
• CognitionCapturer: Decoding Visual Stimuli From Human EEG Signal With Multimodal Information [61.1904164368732]
  We propose CognitionCapturer, a unified framework that fully leverages multimodal data to represent EEG signals.<n>Specifically, CognitionCapturer trains Modality Experts for each modality to extract cross-modal information from the EEG modality.<n>The framework does not require any fine-tuning of the generative models and can be extended to incorporate more modalities.
  arXiv  Detail & Related papers  (2024-12-13T16:27:54Z)
• Visual Neural Decoding via Improved Visual-EEG Semantic Consistency [3.4061238650474657]
  Methods that directly map EEG features to the CLIP embedding space may introduce mapping bias and cause semantic inconsistency. We propose a Visual-EEG Semantic Decouple Framework that explicitly extracts the semantic-related features of these two modalities to facilitate optimal alignment. Our method achieves state-of-the-art results in zero-shot neural decoding tasks.
  arXiv  Detail & Related papers  (2024-08-13T10:16:10Z)
• Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning [2.087148326341881]
  This paper introduces a MUltimodal Similarity-keeping contrastivE learning framework for zero-shot EEG-based image classification. We develop a series of multivariate time-series encoders tailored for EEG signals and assess the efficacy of regularized contrastive EEG-Image pretraining. Our method achieves state-of-the-art performance, with a top-1 accuracy of 19.3% and a top-5 accuracy of 48.8% in 200-way zero-shot image classification.
  arXiv  Detail & Related papers  (2024-06-05T16:42:23Z)
• Alleviating Catastrophic Forgetting in Facial Expression Recognition with Emotion-Centered Models [49.3179290313959]
  The proposed method, emotion-centered generative replay (ECgr), tackles this challenge by integrating synthetic images from generative adversarial networks. ECgr incorporates a quality assurance algorithm to ensure the fidelity of generated images. The experimental results on four diverse facial expression datasets demonstrate that incorporating images generated by our pseudo-rehearsal method enhances training on the targeted dataset and the source dataset.
  arXiv  Detail & Related papers  (2024-04-18T15:28:34Z)
• BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction [7.512223286737468]
  Analyzing and reconstructing visual stimuli from brain signals effectively advances the understanding of human visual system. However, the EEG signals are complex and contain significant noise. This leads to substantial limitations in existing works of visual stimuli reconstruction from EEG. We propose a novel approach called BrainVis to address these challenges.
  arXiv  Detail & Related papers  (2023-12-22T17:49:11Z)
• Learning Robust Deep Visual Representations from EEG Brain Recordings [13.768240137063428]
  This study proposes a two-stage method where the first step is to obtain EEG-derived features for robust learning of deep representations. We demonstrate the generalizability of our feature extraction pipeline across three different datasets using deep-learning architectures. We propose a novel framework to transform unseen images into the EEG space and reconstruct them with approximation.
  arXiv  Detail & Related papers  (2023-10-25T10:26:07Z)
• A Knowledge-Driven Cross-view Contrastive Learning for EEG Representation [48.85731427874065]
  This paper proposes a knowledge-driven cross-view contrastive learning framework (KDC2) to extract effective representations from EEG with limited labels. The KDC2 method creates scalp and neural views of EEG signals, simulating the internal and external representation of brain activity. By modeling prior neural knowledge based on neural information consistency theory, the proposed method extracts invariant and complementary neural knowledge to generate combined representations.
  arXiv  Detail & Related papers  (2023-09-21T08:53:51Z)
• Decoding Natural Images from EEG for Object Recognition [8.411976038504589]
  This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals. We achieve a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in challenging 200-way zero-shot tasks. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios.
  arXiv  Detail & Related papers  (2023-08-25T08:05:37Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships [68.8204255655161]
  We present an interpretable domain grounded solution to recover the activity of several subcortical regions from multichannel EEG data. We recover individual spatial and time-frequency patterns of scalp EEG predictive of the hemodynamic signal in the subcortical nuclei.
  arXiv  Detail & Related papers  (2022-10-23T15:11:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.