Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning

• URL: http://arxiv.org/abs/2602.02784v1
• Date: Mon, 02 Feb 2026 20:40:04 GMT
• Title: Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning
• Abstract summary: Cross-Temporal Attention Fusion (CTAF) is a self-supervised module that learns soft bidirectional alignments between modalities.<n>CTAF yields higher cosine margins for matched pairs and better cross-modal token retrieval within one second.<n>Our contributions are a time-aware fusion mechanism that directly models correspondence, an alignment-driven self-supervised objective tailored to EEG and physiology, and an evaluation protocol that measures alignment quality itself.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study multimodal affect modeling when EEG and peripheral physiology are asynchronous, which most fusion methods ignore or handle with costly warping. We propose Cross-Temporal Attention Fusion (CTAF), a self-supervised module that learns soft bidirectional alignments between modalities and builds a robust clip embedding using time-aware cross attention, a lightweight fusion gate, and alignment-regularized contrastive objectives with optional weak supervision. On the K-EmoCon dataset, under leave-one-out cross-validation evaluation, CTAF yields higher cosine margins for matched pairs and better cross-modal token retrieval within one second, and it is competitive with the baseline on three-bin accuracy and macro-F1 while using few labels. Our contributions are a time-aware fusion mechanism that directly models correspondence, an alignment-driven self-supervised objective tailored to EEG and physiology, and an evaluation protocol that measures alignment quality itself. Our approach accounts for the coupling between the central and autonomic nervous systems in psychophysiological time series. These results indicate that CTAF is a strong step toward label-efficient, generalizable EEG-peripheral fusion under temporal asynchrony.

Related papers

• MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment [4.682152272555128]
  This paper presents MUPAtextsuperscript2E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone.<n> Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion.<n>Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride$30$ achieves the highest validation performance and a test accuracy of $70.07%$.
  arXiv  Detail & Related papers  (2026-08-17T01:26:47Z)
• Multimodal Wearable-Based Olfactory-Induced Emotion Recognition in Arousal-Valence Dimensions [15.21606334710936]
  Olfaction acts as a non-intrusive and lightweight pathway that directly engages the brain s affective circuitry and emotional modulation.<n>Current olfactory emotion research has two key limitations.<n>It over-emphasises the valence dimension while neglecting arousal.
  arXiv  Detail & Related papers  (2026-07-24T01:42:27Z)
• Moving Beyond Functional Connectivity: Time-Series Modeling for fMRI-Based Brain Disorder Classification [8.837732238971187]
  Functional magnetic resonance imaging (fMRI) enables non-invasive brain disorder classification by capturing blood-oxygen-level-dependent (BOLD) signals.<n>Most existing methods rely on functional connectivity (FC) via Pearson correlation.<n>We benchmark state-of-the-art temporal models on raw BOLD signals across five public datasets.
  arXiv  Detail & Related papers  (2026-02-09T04:42:42Z)
• TokaMark: A Comprehensive Benchmark for MAST Tokamak Plasma Models [56.94569090844015]
  TokaMark is a structured benchmark to evaluate AI models on real experimental data collected from the Mega Ampere Spherical Tokamak (MAST)<n>TokaMark aims to accelerate progress in data-driven AI-based plasma modeling, contributing to the broader goal of achieving sustainable and stable fusion energy.
  arXiv  Detail & Related papers  (2026-02-05T16:49:44Z)
• Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech [51.14752758616364]
  Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments.<n>We propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework.<n>The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.
  arXiv  Detail & Related papers  (2025-10-05T09:32:12Z)
• Explainable Deep Neural Network for Multimodal ECG Signals: Intermediate vs Late Fusion [1.1344265020822928]
  Multimodal deep neural networks (MDNN) have the capability of integrating diverse data domains and offer a promising solution for robust and accurate predictions.<n>This study investigates the comparative effectiveness of intermediate and late fusion strategies using ECG signals.
  arXiv  Detail & Related papers  (2025-08-06T16:26:46Z)
• MHSA: A Multi-scale Hypergraph Network for Mild Cognitive Impairment Detection via Synchronous and Attentive Fusion [4.526574526136158]
  A Multi-scale Hypergraph Network for MCI Detection via Synchronous and Attentive Fusion is presented.<n>Our approach employs the Phase-Locking Value (PLV) to calculate the phase synchronization relationship in the spectrum domain of regions of interest.<n>We structure the PLV coefficients dynamically adjust strategy, and the dynamic hypergraph is modelled based on a comprehensive temporal-spectrum fusion matrix.
  arXiv  Detail & Related papers  (2024-12-11T02:59:57Z)
• CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis [50.56875995511431]
  We introduce a Cross-Modal Temporal Pattern Discovery (CTPD) framework, designed to efficiently extract meaningful cross-modal temporal patterns from multimodal EHR data.<n>Our approach introduces shared initial temporal pattern representations which are refined using slot attention to generate temporal semantic embeddings.
  arXiv  Detail & Related papers  (2024-11-01T15:54:07Z)
• PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation [51.509573838103854]
  We propose a semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation. Our PMT generates high-fidelity pseudo labels by learning robust and diverse features in the training process. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches.
  arXiv  Detail & Related papers  (2024-09-08T15:02:25Z)
• DA-Flow: Dual Attention Normalizing Flow for Skeleton-based Video Anomaly Detection [52.74152717667157]
  We propose a lightweight module called Dual Attention Module (DAM) for capturing cross-dimension interaction relationships in-temporal skeletal data. It employs the frame attention mechanism to identify the most significant frames and the skeleton attention mechanism to capture broader relationships across fixed partitions with minimal parameters and flops.
  arXiv  Detail & Related papers  (2024-06-05T06:18:03Z)
• MMA-RNN: A Multi-level Multi-task Attention-based Recurrent Neural Network for Discrimination and Localization of Atrial Fibrillation [1.8037893225125925]
  This paper proposes the Multi-level Multi-task Attention-based Recurrent Neural Network for three-class discrimination on patients and localization of the exact timing of AF episodes. The model is designed as an end-to-end framework to enhance information interaction and reduce error accumulation.
  arXiv  Detail & Related papers  (2023-02-07T19:59:55Z)
• ECG Heartbeat Classification Using Multimodal Fusion [13.524306011331303]
  We propose two computationally efficient multimodal fusion frameworks for ECG heart beat classification. In MFF, we extracted features from penultimate layer of CNNs and fused them to get unique and interdependent information. We achieved classification accuracy of 99.7% and 99.2% on arrhythmia and MI classification, respectively.
  arXiv  Detail & Related papers  (2021-07-21T03:48:35Z)
• Detecting Pancreatic Ductal Adenocarcinoma in Multi-phase CT Scans via Alignment Ensemble [77.5625174267105]
  Pancreatic ductal adenocarcinoma (PDAC) is one of the most lethal cancers among the population. Multiple phases provide more information than single phase, but they are unaligned and inhomogeneous in texture. We suggest an ensemble of all these alignments as a promising way to boost the performance of PDAC detection.
  arXiv  Detail & Related papers  (2020-03-18T19:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.