Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation

• URL: http://arxiv.org/abs/2412.05296v1
• Date: Sun, 24 Nov 2024 16:04:03 GMT
• Title: Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation
• Authors: Joonwoo Kwon, Heehwan Wang, Jinwoo Lee, Sooyoung Kim, Shinjae Yoo, Yuewei Lin, Jiook Cha,
• Abstract summary: We introduce RecallAffectiveMemory, a novel task designed to reconstruct autobiographical memories through audio-visual generation guided by affect extracted from electroencephalogram (EEG) signals.<n>We present the EEG-AffectiveMemory dataset, which encompasses textual descriptions, visuals, music, and EEG recordings collected during memory recall from nine participants.
• Score: 7.67506894657724
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we introduce RecallAffectiveMemory, a novel task designed to reconstruct autobiographical memories through audio-visual generation guided by affect extracted from electroencephalogram (EEG) signals. To support this pioneering task, we present the EEG-AffectiveMemory dataset, which encompasses textual descriptions, visuals, music, and EEG recordings collected during memory recall from nine participants. Furthermore, we propose RYM (Recall Your Memory), a three-stage framework for generating synchronized audio-visual contents while maintaining dynamic personal memory affect trajectories. Experimental results indicate that our method can faithfully reconstruct affect-contextualized audio-visual memory across all subjects, both qualitatively and quantitatively, with participants reporting strong affective concordance between their recalled memories and the generated content. Our approaches advance affect decoding research and its practical applications in personalized media creation via neural-based affect comprehension.

Related papers

• Latent Structured Hopfield Network for Semantic Association and Retrieval [52.634915010996835]
  Episodic memory enables humans to recall past experiences by associating semantic elements such as objects, locations, and time into coherent event representations.<n>We propose the Latent Structured Hopfield Network (LSHN), a framework that integrates continuous Hopfield attractor dynamics into an autoencoder architecture.<n>Unlike traditional Hopfield networks, our model is trained end-to-end with gradient descent, achieving scalable and robust memory retrieval.
  arXiv  Detail & Related papers  (2025-06-02T04:24:36Z)
• On Memory Construction and Retrieval for Personalized Conversational Agents [69.46887405020186]
  We propose SeCom, a method that constructs the memory bank at segment level by introducing a conversation segmentation model. Experimental results show that SeCom exhibits a significant performance advantage over baselines on long-term conversation benchmarks LOCOMO and Long-MT-Bench+.
  arXiv  Detail & Related papers  (2025-02-08T14:28:36Z)
• Decoding the Echoes of Vision from fMRI: Memory Disentangling for Past Semantic Information [26.837527605133]
  This study investigates the capacity of working memory to retain past information under continuous visual stimuli. We propose a new task Memory Disentangling, which aims to extract and decode past information from fMRI signals.
  arXiv  Detail & Related papers  (2024-09-30T15:51:06Z)
• In-Memory Learning: A Declarative Learning Framework for Large Language Models [56.62616975119192]
  We propose a novel learning framework that allows agents to align with their environment without relying on human-labeled data. This entire process transpires within the memory components and is implemented through natural language. We demonstrate the effectiveness of our framework and provide insights into this problem.
  arXiv  Detail & Related papers  (2024-03-05T08:25:11Z)
• BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction [7.512223286737468]
  Analyzing and reconstructing visual stimuli from brain signals effectively advances the understanding of human visual system. However, the EEG signals are complex and contain significant noise. This leads to substantial limitations in existing works of visual stimuli reconstruction from EEG. We propose a novel approach called BrainVis to address these challenges.
  arXiv  Detail & Related papers  (2023-12-22T17:49:11Z)
• Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues [80.53407593586411]
  We focus on leveraging facial cues beyond the lip region for robust Audio-Visual Speech Enhancement (AVSE) We propose a Dual Attention Cooperative Framework, DualAVSE, to ignore speech-unrelated information, capture speech-related information with facial cues, and dynamically integrate it with the audio signal for AVSE.
  arXiv  Detail & Related papers  (2023-11-24T04:30:31Z)
• Saliency-Guided Hidden Associative Replay for Continual Learning [13.551181595881326]
  Continual Learning is a burgeoning domain in next-generation AI, focusing on training neural networks over a sequence of tasks akin to human learning. This paper presents the Saliency Guided Hidden Associative Replay for Continual Learning. This novel framework synergizes associative memory with replay-based strategies. SHARC primarily archives salient data segments via sparse memory encoding.
  arXiv  Detail & Related papers  (2023-10-06T15:54:12Z)
• Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models [65.08133391009838]
  generative process of Diffusion Models (DMs) has recently set state-of-the-art on many AI generation benchmarks. We introduce a novel perspective to describe DMs using the mathematical language of memory retrieval from the field of energy-based Associative Memories (AMs) We present a growing body of evidence that records DMs exhibiting empirical behavior we would expect from AMs, and conclude by discussing research opportunities that are revealed by understanding DMs as a form of energy-based memory.
  arXiv  Detail & Related papers  (2023-09-28T17:57:09Z)
• Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals [27.92796103924193]
  We propose a comprehensive pipeline, named NeuroImagen, for reconstructing visual stimuli images from EEG signals. We incorporate a novel multi-level perceptual information decoding to draw multi-grained outputs from the given EEG data.
  arXiv  Detail & Related papers  (2023-07-27T12:54:16Z)
• Improving Image Recognition by Retrieving from Web-Scale Image-Text Data [68.63453336523318]
  We introduce an attention-based memory module, which learns the importance of each retrieved example from the memory. Compared to existing approaches, our method removes the influence of the irrelevant retrieved examples, and retains those that are beneficial to the input query. We show that it achieves state-of-the-art accuracies in ImageNet-LT, Places-LT and Webvision datasets.
  arXiv  Detail & Related papers  (2023-04-11T12:12:05Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• Mind Reader: Reconstructing complex images from brain activities [16.78619734818198]
  We focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects or simple shapes, our work aims to reconstruct image stimuli rich in semantics. We find that incorporating an additional text modality is beneficial for the reconstruction problem compared to directly translating brain signals to images.
  arXiv  Detail & Related papers  (2022-09-30T06:32:46Z)
• Audio-visual multi-channel speech separation, dereverberation and recognition [70.34433820322323]
  This paper proposes an audio-visual multi-channel speech separation, dereverberation and recognition approach. The advantage of the additional visual modality over using audio only is demonstrated on two neural dereverberation approaches. Experiments conducted on the LRS2 dataset suggest that the proposed audio-visual multi-channel speech separation, dereverberation and recognition system outperforms the baseline.
  arXiv  Detail & Related papers  (2022-04-05T04:16:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.