Making Your Dreams A Reality: Decoding the Dreams into a Coherent Video Story from fMRI Signals

• URL: http://arxiv.org/abs/2501.09350v1
• Date: Thu, 16 Jan 2025 08:03:49 GMT
• Title: Making Your Dreams A Reality: Decoding the Dreams into a Coherent Video Story from fMRI Signals
• Authors: Yanwei Fu, Jianxiong Gao, Baofeng Yang, Jianfeng Feng,
• Abstract summary: This paper studies the brave new idea for Multimedia community, and proposes a novel framework to convert dreams into coherent video narratives. Recent advancements in brain imaging, particularly functional magnetic resonance imaging (fMRI), have provided new ways to explore the neural basis of dreaming. By combining subjective dream experiences with objective neurophysiological data, we aim to understand the visual aspects of dreams and create complete video narratives.
• Score: 46.90535445975669
• License:
• Abstract: This paper studies the brave new idea for Multimedia community, and proposes a novel framework to convert dreams into coherent video narratives using fMRI data. Essentially, dreams have intrigued humanity for centuries, offering glimpses into our subconscious minds. Recent advancements in brain imaging, particularly functional magnetic resonance imaging (fMRI), have provided new ways to explore the neural basis of dreaming. By combining subjective dream experiences with objective neurophysiological data, we aim to understand the visual aspects of dreams and create complete video narratives. Our process involves three main steps: reconstructing visual perception, decoding dream imagery, and integrating dream stories. Using innovative techniques in fMRI analysis and language modeling, we seek to push the boundaries of dream research and gain deeper insights into visual experiences during sleep. This technical report introduces a novel approach to visually decoding dreams using fMRI signals and weaving dream visuals into narratives using language models. We gather a dataset of dreams along with descriptions to assess the effectiveness of our framework.

Related papers

• Neuro-3D: Towards 3D Visual Decoding from EEG Signals [49.502364730056044]
  We introduce a new neuroscience task: decoding 3D visual perception from EEG signals. We first present EEG-3D, a dataset featuring multimodal analysis data and EEG recordings from 12 subjects viewing 72 categories of 3D objects rendered in both videos and images. We propose Neuro-3D, a 3D visual decoding framework based on EEG signals.
  arXiv  Detail & Related papers  (2024-11-19T05:52:17Z)
• Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion [0.11249583407496218]
  This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across multimodal representationsof brain activity by leveraging contrastive learning. We used electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. Our framework's capabilities are demonstrated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities.
  arXiv  Detail & Related papers  (2024-11-14T12:27:27Z)
• MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models [57.47799823804519]
  We are inspired by how musicians compose music not just from a movie script, but also through visualizations. We propose MeLFusion, a model that can effectively use cues from a textual description and the corresponding image to synthesize music. Our exhaustive experimental evaluation suggests that adding visual information to the music synthesis pipeline significantly improves the quality of generated music.
  arXiv  Detail & Related papers  (2024-06-07T06:38:59Z)
• Brain3D: Generating 3D Objects from fMRI [76.41771117405973]
  We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject. We show that our model captures the distinct functionalities of each region of human vision system. Preliminary evaluations indicate that Brain3D can successfully identify the disordered brain regions in simulated scenarios.
  arXiv  Detail & Related papers  (2024-05-24T06:06:11Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI [12.203617776046169]
  We introduce a novel framework named Brainformer to analyze fMRI patterns in the human perception system. This work introduces a prospective approach to transferring knowledge from human perception to neural networks.
  arXiv  Detail & Related papers  (2023-11-30T22:39:23Z)
• Dream Content Discovery from Reddit with an Unsupervised Mixed-Method Approach [0.8127745323109788]
  We developed a new, data-driven mixed-method approach for identifying topics in free-form dream reports through natural language processing. We tested this method on 44,213 dream reports from Reddit's r/Dreams subreddit. Our method can find unique patterns in different dream types, understand topic importance and connections, and observe changes in collective dream experiences over time and around major events.
  arXiv  Detail & Related papers  (2023-07-09T13:24:58Z)
• DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embedding [6.497816402045099]
  We propose fMRI captioning, where captions are generated based on fMRI data to gain insight into visual perception. DreamCatcher consists of the Representation Space (RSE) and the RevEmbedding Decoder, which transform fMRI into a latent space vectors generate captions. fMRI-based captioning has diverse applications, including understanding neural mechanisms, Human-Computer Interaction, and enhancing learning and training processes.
  arXiv  Detail & Related papers  (2023-06-16T07:55:20Z)
• Memory semantization through perturbed and adversarial dreaming [0.7874708385247353]
  We propose that rapid-eye-movement (REM) dreaming is essential for efficient memory semantization. We implement a cortical architecture with hierarchically organized feedforward and feedback pathways, inspired by generative adversarial networks (GANs) Our results suggest that adversarial dreaming during REM sleep is essential for extracting memory contents, while dreaming during NREM sleep improves the robustness of the latent representation to noisy sensory inputs.
  arXiv  Detail & Related papers  (2021-09-09T13:31:13Z)
• Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans [56.63912568777483]
  This paper addresses the challenge of novel view synthesis for a human performer from a very sparse set of camera views. We propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh. Experiments on ZJU-MoCap show that our approach outperforms prior works by a large margin in terms of novel view synthesis quality.
  arXiv  Detail & Related papers  (2020-12-31T18:55:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.