DynaMind: Reconstructing Dynamic Visual Scenes from EEG by Aligning Temporal Dynamics and Multimodal Semantics to Guided Diffusion

• URL: http://arxiv.org/abs/2509.01177v1
• Date: Mon, 01 Sep 2025 06:52:08 GMT
• Title: DynaMind: Reconstructing Dynamic Visual Scenes from EEG by Aligning Temporal Dynamics and Multimodal Semantics to Guided Diffusion
• Authors: Junxiang Liu, Junming Lin, Jiangtong Li, Jie Li,
• Abstract summary: We introduce DynaMind, a novel framework that reconstructs video by jointly modeling neural dynamics and semantic features.<n>On the SEED-DV dataset, DynaMind sets a new state-of-the-art (SOTA), boosting reconstructed video accuracies by 12.5 and 10.3 percentage points.<n>This marks a critical advancement, bridging the gap between neural dynamics and high-fidelity visual semantics.
• Score: 10.936858717759156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstruction dynamic visual scenes from electroencephalography (EEG) signals remains a primary challenge in brain decoding, limited by the low spatial resolution of EEG, a temporal mismatch between neural recordings and video dynamics, and the insufficient use of semantic information within brain activity. Therefore, existing methods often inadequately resolve both the dynamic coherence and the complex semantic context of the perceived visual stimuli. To overcome these limitations, we introduce DynaMind, a novel framework that reconstructs video by jointly modeling neural dynamics and semantic features via three core modules: a Regional-aware Semantic Mapper (RSM), a Temporal-aware Dynamic Aligner (TDA), and a Dual-Guidance Video Reconstructor (DGVR). The RSM first utilizes a regional-aware encoder to extract multimodal semantic features from EEG signals across distinct brain regions, aggregating them into a unified diffusion prior. In the mean time, the TDA generates a dynamic latent sequence, or blueprint, to enforce temporal consistency between the feature representations and the original neural recordings. Together, guided by the semantic diffusion prior, the DGVR translates the temporal-aware blueprint into a high-fidelity video reconstruction. On the SEED-DV dataset, DynaMind sets a new state-of-the-art (SOTA), boosting reconstructed video accuracies (video- and frame-based) by 12.5 and 10.3 percentage points, respectively. It also achieves a leap in pixel-level quality, showing exceptional visual fidelity and temporal coherence with a 9.4% SSIM improvement and a 19.7% FVMD reduction. This marks a critical advancement, bridging the gap between neural dynamics and high-fidelity visual semantics.

Related papers

• SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance [52.34513874272676]
  We introduce SemVideo, a novel fMRI-to-video reconstruction framework guided by hierarchical semantic information.<n>At the core of SemVideo is SemMiner, a hierarchical guidance module that constructs three levels of semantic cues from the original video stimulus.<n>We show that SemVideo achieves superior performance in both semantic alignment and temporal consistency, setting a new state-of-the-art in fMRI-to-video reconstruction.
  arXiv  Detail & Related papers  (2026-02-25T11:47:09Z)
• Contrastive and Multi-Task Learning on Noisy Brain Signals with Nonlinear Dynamical Signatures [5.37454752035459]
  We introduce a two-stage multitask learning framework for analyzing EEG signals.<n>In the first stage, a denoising autoencoder is trained to suppress artifacts and stabilize temporal dynamics.<n>In the second stage, a multitask architecture processes these denoised signals to achieve three objectives.
  arXiv  Detail & Related papers  (2026-01-13T13:36:38Z)
• Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction [65.67001243986981]
  We propose MindHier, a coarse-to-fine fMRI-to-image reconstruction framework built on scale-wise autoregressive modeling.<n>MindHier achieves superior semantic fidelity, 4.67x faster inference, and more deterministic results than the diffusion-based baselines.
  arXiv  Detail & Related papers  (2025-10-25T15:40:07Z)
• Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillation [21.738308923180767]
  We present DISCOVR, a self-supervised dual branch framework for cardiac ultrasound video representation learning.<n>DISCOVR combines a clustering-based video encoder that models temporal dynamics with an online image encoder that extracts fine-grained spatial semantics.
  arXiv  Detail & Related papers  (2025-06-13T13:36:33Z)
• Pursuing Temporal-Consistent Video Virtual Try-On via Dynamic Pose Interaction [142.66410908560582]
  Video virtual try-on aims to seamlessly dress a subject in a video physique with a specific garment.<n>We propose Dynamic Pose Interaction Diffusion Models (DPIDM) to leverage diffusion models to delve into dynamic pose interactions for video virtual try-on.<n>DPIDM capitalizes on a temporal regularized attention loss between consecutive frames to enhance temporal consistency.
  arXiv  Detail & Related papers  (2025-05-22T17:52:34Z)
• Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRI [3.0450307343472405]
  We introduce Dynadiff, a new single-stage diffusion model designed for reconstructing images from dynamically evolving fMRI recordings.<n>Our model outperforms state-of-the-art models on time-resolved fMRI signals, especially on high-level semantic image reconstruction metrics.<n>Overall, this work lays the foundation for time-resolved brain-to-image decoding.
  arXiv  Detail & Related papers  (2025-05-20T16:14:37Z)
• Temporal-Consistent Video Restoration with Pre-trained Diffusion Models [51.47188802535954]
  Video restoration (VR) aims to recover high-quality videos from degraded ones.<n>Recent zero-shot VR methods using pre-trained diffusion models (DMs) suffer from approximation errors during reverse diffusion and insufficient temporal consistency.<n>We present a novel a Posterior Maximum (MAP) framework that directly parameterizes video frames in the seed space of DMs, eliminating approximation errors.
  arXiv  Detail & Related papers  (2025-03-19T03:41:56Z)
• EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting [95.44545809256473]
  EgoGaussian is a method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We show significant improvements in terms of both dynamic object and background reconstruction quality compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-06-28T10:39:36Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity [13.04953215936574]
  We propose a two-stage model named Mind-Animator to reconstruct human dynamic vision from brain activity.<n>During the fMRI-to-feature stage, we decouple semantic, structure, and motion features from fMRI.<n>In the feature-to-video stage, these features are integrated into videos using an inflated Stable Diffusion.
  arXiv  Detail & Related papers  (2024-05-06T08:56:41Z)
• NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties [23.893490180665996]
  We introduce NeuroCine, a novel dual-phase framework to targeting the inherent challenges of decoding fMRI data. tested on a publicly available fMRI dataset, our method shows promising results. Our attention analysis suggests that the model aligns with existing brain structures and functions, indicating its biological plausibility and interpretability.
  arXiv  Detail & Related papers  (2024-02-02T17:34:25Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks [68.93429034530077]
  We propose dynamics-aware implicit generative adversarial network (DIGAN) for video generation. We show that DIGAN can be trained on 128 frame videos of 128x128 resolution, 80 frames longer than the 48 frames of the previous state-of-the-art method.
  arXiv  Detail & Related papers  (2022-02-21T23:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.