Related papers: Latent Representation Learning for Multimodal Brain Activity Translation

Latent Representation Learning for Multimodal Brain Activity Translation

URL: http://arxiv.org/abs/2409.18462v1
Date: Fri, 27 Sep 2024 05:50:29 GMT
Title: Latent Representation Learning for Multimodal Brain Activity Translation
Authors: Arman Afrasiyabi, Dhananjay Bhaskar, Erica L. Busch, Laurent Caplette, Rahul Singh, Guillaume Lajoie, Nicholas B. Turk-Browne, Smita Krishnaswamy,
Abstract summary: We present the Spatiotemporal Alignment of Multimodal Brain Activity (SAMBA) framework, which bridges the spatial and temporal resolution gaps across modalities. SAMBA introduces a novel attention-based wavelet decomposition for spectral filtering of electrophysiological recordings. We show that the training of SAMBA, aside from achieving translation, also learns a rich representation of brain information processing.
Score: 14.511112110420271
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Neuroscience employs diverse neuroimaging techniques, each offering distinct insights into brain activity, from electrophysiological recordings such as EEG, which have high temporal resolution, to hemodynamic modalities such as fMRI, which have increased spatial precision. However, integrating these heterogeneous data sources remains a challenge, which limits a comprehensive understanding of brain function. We present the Spatiotemporal Alignment of Multimodal Brain Activity (SAMBA) framework, which bridges the spatial and temporal resolution gaps across modalities by learning a unified latent space free of modality-specific biases. SAMBA introduces a novel attention-based wavelet decomposition for spectral filtering of electrophysiological recordings, graph attention networks to model functional connectivity between functional brain units, and recurrent layers to capture temporal autocorrelations in brain signal. We show that the training of SAMBA, aside from achieving translation, also learns a rich representation of brain information processing. We showcase this classify external stimuli driving brain activity from the representation learned in hidden layers of SAMBA, paving the way for broad downstream applications in neuroscience research and clinical contexts.