CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation

• URL: http://arxiv.org/abs/2408.00777v1
• Date: Tue, 16 Jul 2024 11:31:38 GMT
• Title: CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation
• Authors: Weiheng Yao, Shuqiang Wang,
• Abstract summary: This paper proposes the Condition-Aligned Temporal Diffusion (CATD) framework for end-to-end cross-modal synthesis of neuroimaging. The proposed framework establishes a new paradigm for cross-modal synthesis of neuroimaging. It shows promise in medical applications such as improving Parkinson's disease prediction and identifying abnormal brain regions.
• Score: 6.682531937245544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal neuroimaging analysis is crucial for a comprehensive understanding of brain function and pathology, as it allows for the integration of different imaging techniques, thus overcoming the limitations of individual modalities. However, the high costs and limited availability of certain modalities pose significant challenges. To address these issues, this paper proposed the Condition-Aligned Temporal Diffusion (CATD) framework for end-to-end cross-modal synthesis of neuroimaging, enabling the generation of functional magnetic resonance imaging (fMRI)-detected Blood Oxygen Level Dependent (BOLD) signals from more accessible Electroencephalography (EEG) signals. By constructing Conditionally Aligned Block (CAB), heterogeneous neuroimages are aligned into a potential space, achieving a unified representation that provides the foundation for cross-modal transformation in neuroimaging. The combination with the constructed Dynamic Time-Frequency Segmentation (DTFS) module also enables the use of EEG signals to improve the temporal resolution of BOLD signals, thus augmenting the capture of the dynamic details of the brain. Experimental validation demonstrated the effectiveness of the framework in improving the accuracy of neural activity prediction, identifying abnormal brain regions, and enhancing the temporal resolution of BOLD signals. The proposed framework establishes a new paradigm for cross-modal synthesis of neuroimaging by unifying heterogeneous neuroimaging data into a potential representation space, showing promise in medical applications such as improving Parkinson's disease prediction and identifying abnormal brain regions.

Related papers

• NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping [9.423808859117122]
  We introduce NeuroBOLT, i.e., Neuro-to-BOLD Transformer, to translate raw EEG data to fMRI activity signals across the brain. Our experiments demonstrate that NeuroBOLT effectively reconstructs unseen resting-state fMRI signals from primary sensory, high-level cognitive areas, and deep subcortical brain regions.
  arXiv  Detail & Related papers  (2024-10-07T02:47:55Z)
• Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation [56.34634121544929]
  In this study, we first construct the brain-effective network via the dynamic causal model. We then introduce an interpretable graph learning framework termed Spatio-Temporal Embedding ODE (STE-ODE) This framework incorporates specifically designed directed node embedding layers, aiming at capturing the dynamic interplay between structural and effective networks.
  arXiv  Detail & Related papers  (2024-05-21T20:37:07Z)
• BrainODE: Dynamic Brain Signal Analysis via Graph-Aided Neural Ordinary Differential Equations [67.79256149583108]
  We propose a novel model called BrainODE to achieve continuous modeling of dynamic brain signals. By learning latent initial values and neural ODE functions from irregular time series, BrainODE effectively reconstructs brain signals at any time point.
  arXiv  Detail & Related papers  (2024-04-30T10:53:30Z)
• Leveraging sinusoidal representation networks to predict fMRI signals from EEG [3.3121941932506473]
  We propose a novel architecture that can predict fMRI signals directly from multi-channel EEG without explicit feature engineering. Our model achieves this by implementing a Sinusoidal Representation Network (SIREN) to learn frequency information in brain dynamics. We evaluate our model using a simultaneous EEG-fMRI dataset with 8 subjects and investigate its potential for predicting subcortical fMRI signals.
  arXiv  Detail & Related papers  (2023-11-06T03:16:18Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• DiffGAN-F2S: Symmetric and Efficient Denoising Diffusion GANs for Structural Connectivity Prediction from Brain fMRI [15.40111168345568]
  It is challenging to bridge the reliable non-linear mapping relations between structural connectivity (SC) and functional magnetic resonance imaging (fMRI) A novel diffusision generative adversarial network-based fMRI-to-SC model is proposed to predict SC from brain fMRI in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-28T06:55:50Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• Patched Diffusion Models for Unsupervised Anomaly Detection in Brain MRI [55.78588835407174]
  We propose a method that reformulates the generation task of diffusion models as a patch-based estimation of healthy brain anatomy. We evaluate our approach on data of tumors and multiple sclerosis lesions and demonstrate a relative improvement of 25.1% compared to existing baselines.
  arXiv  Detail & Related papers  (2023-03-07T09:40:22Z)
• fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships [68.8204255655161]
  We present an interpretable domain grounded solution to recover the activity of several subcortical regions from multichannel EEG data. We recover individual spatial and time-frequency patterns of scalp EEG predictive of the hemodynamic signal in the subcortical nuclei.
  arXiv  Detail & Related papers  (2022-10-23T15:11:37Z)
• Inferring Brain Dynamics via Multimodal Joint Graph Representation EEG-fMRI [0.0]
  We show that multi-modeling methods can provide new insights into the neural analysis of brain components that are not possible when each modality is acquired separately. The joint representations of different modalities is a robust model to analyze simultaneously acquired electroencephalography and magnetic resonance imaging (EEG-fMRI) We outline the correlations of several different media in time from one source with graph-based deep learning methods.
  arXiv  Detail & Related papers  (2022-01-21T15:39:48Z)
• Characterization Multimodal Connectivity of Brain Network by Hypergraph GAN for Alzheimer's Disease Analysis [30.99183477161096]
  multimodal neuroimaging data to characterize brain network is currently an advanced technique for Alzheimer's disease(AD) Analysis. We propose a novel Hypergraph Generative Adversarial Networks(HGGAN) to generate multimodal connectivity of Brain Network from rs-fMRI combination with DTI.
  arXiv  Detail & Related papers  (2021-07-21T09:02:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.