DreamDiffusion: Generating High-Quality Images from Brain EEG Signals

• URL: http://arxiv.org/abs/2306.16934v2
• Date: Fri, 30 Jun 2023 10:46:54 GMT
• Title: DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
• Authors: Yunpeng Bai, Xintao Wang, Yan-pei Cao, Yixiao Ge, Chun Yuan, Ying Shan
• Abstract summary: DreamDiffusion is a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals. The proposed method overcomes the challenges of using EEG signals for image generation, such as noise, limited information, and individual differences.
• Score: 42.30835251506628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces DreamDiffusion, a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals, without the need to translate thoughts into text. DreamDiffusion leverages pre-trained text-to-image models and employs temporal masked signal modeling to pre-train the EEG encoder for effective and robust EEG representations. Additionally, the method further leverages the CLIP image encoder to provide extra supervision to better align EEG, text, and image embeddings with limited EEG-image pairs. Overall, the proposed method overcomes the challenges of using EEG signals for image generation, such as noise, limited information, and individual differences, and achieves promising results. Quantitative and qualitative results demonstrate the effectiveness of the proposed method as a significant step towards portable and low-cost ``thoughts-to-image'', with potential applications in neuroscience and computer vision. The code is available here \url{https://github.com/bbaaii/DreamDiffusion}.

Related papers

• BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance [14.003870853594972]
  We introduce BrainDreamer, a novel end-to-end language-guided generative framework. BrainDreamer can mimic human reasoning and generate high-quality images from electroencephalogram (EEG) brain signals. Our method is superior in its capacity to eliminate the noise introduced by non-invasive EEG data acquisition.
  arXiv  Detail & Related papers  (2024-09-21T05:16:31Z)
• Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models [4.933734706786783]
  EEG is a low-cost, non-invasive, and portable neuroimaging technique. EEG presents inherent challenges due to its low spatial resolution and susceptibility to noise and artifacts. We propose a framework based on the ControlNet adapter for conditioning a latent diffusion model through EEG signals.
  arXiv  Detail & Related papers  (2024-09-17T19:07:13Z)
• Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learning [2.087148326341881]
  This paper introduces a MUltimodal Similarity-keeping contrastivE learning framework for zero-shot EEG-based image classification. We develop a series of multivariate time-series encoders tailored for EEG signals and assess the efficacy of regularized contrastive EEG-Image pretraining. Our method achieves state-of-the-art performance, with a top-1 accuracy of 19.3% and a top-5 accuracy of 48.8% in 200-way zero-shot image classification.
  arXiv  Detail & Related papers  (2024-06-05T16:42:23Z)
• RIGID: A Training-free and Model-Agnostic Framework for Robust AI-Generated Image Detection [60.960988614701414]
  RIGID is a training-free and model-agnostic method for robust AI-generated image detection. RIGID significantly outperforms existing trainingbased and training-free detectors.
  arXiv  Detail & Related papers  (2024-05-30T14:49:54Z)
• Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [65.7968515029306]
  We propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for Pose-Guided Person Image Synthesis (PGPIS) A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt.
  arXiv  Detail & Related papers  (2024-02-28T06:07:07Z)
• Learning Robust Deep Visual Representations from EEG Brain Recordings [13.768240137063428]
  This study proposes a two-stage method where the first step is to obtain EEG-derived features for robust learning of deep representations. We demonstrate the generalizability of our feature extraction pipeline across three different datasets using deep-learning architectures. We propose a novel framework to transform unseen images into the EEG space and reconstruct them with approximation.
  arXiv  Detail & Related papers  (2023-10-25T10:26:07Z)
• A Knowledge-Driven Cross-view Contrastive Learning for EEG Representation [48.85731427874065]
  This paper proposes a knowledge-driven cross-view contrastive learning framework (KDC2) to extract effective representations from EEG with limited labels. The KDC2 method creates scalp and neural views of EEG signals, simulating the internal and external representation of brain activity. By modeling prior neural knowledge based on neural information consistency theory, the proposed method extracts invariant and complementary neural knowledge to generate combined representations.
  arXiv  Detail & Related papers  (2023-09-21T08:53:51Z)
• Decoding Natural Images from EEG for Object Recognition [8.411976038504589]
  This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals. We achieve a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in challenging 200-way zero-shot tasks. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios.
  arXiv  Detail & Related papers  (2023-08-25T08:05:37Z)
• Semantic Image Synthesis via Diffusion Models [159.4285444680301]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• Harmonizing Pathological and Normal Pixels for Pseudo-healthy Synthesis [68.5287824124996]
  We present a new type of discriminator, the segmentor, to accurately locate the lesions and improve the visual quality of pseudo-healthy images. We apply the generated images into medical image enhancement and utilize the enhanced results to cope with the low contrast problem. Comprehensive experiments on the T2 modality of BraTS demonstrate that the proposed method substantially outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T08:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.