RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection

• URL: http://arxiv.org/abs/2609.11372v1
• Date: Thu, 10 Sep 2026 11:08:15 GMT
• Title: RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection
• Abstract summary: Auditory attention decoding (AAD) identifies the attended speaker from physiological signals.<n>EEG is the dominant modality for AAD but provides incomplete evidence in naturalistic audio-visual scenes.<n>We propose RAMamba-Net, a reliability-aware Mamba-based multimodal fusion network for AAD.
• Score: 17.489739821644598
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Auditory attention decoding (AAD) identifies the attended speaker from physiological signals, supporting neuro-steered hearing devices and natural human-machine interaction. Electroencephalography (EEG) is the dominant modality for AAD but provides incomplete evidence in naturalistic audio-visual scenes, motivating EEG and electrooculography (EOG) fusion. Existing approaches remain limited by weak cross-modal interaction, inefficient temporal modeling, and low robustness to sample variations. To address the limitations, we propose RAMamba-Net, a reliability-aware Mamba-based multimodal fusion network for AAD. RAMamba-Net employs a Mamba-enhanced band-aware convolutional Transformer to capture band-specific EEG patterns and long-range temporal dynamics. A dual-branch temporal-spatial encoder models EOG temporal and inter-channel dependencies. Cross-modal attention enables explicit modality interaction. Then, a reliability-aware module is introduced to estimate sample-wise modality weights for feature and prediction consistency, thereby enhancing multimodal fusion. Experiments on two AAD benchmarks demonstrate that RAMamba-Net effectively exploits complementary EEG-EOG information, yielding accuracy gains of 5.76% over unimodal baselines, together with more robust decoding and discriminative representations. Further analyses show that explicit cross-modal interaction improves multimodal alignment, while the reliability-aware module suppresses unreliable modality evidence and is robust to signal perturbation and parameter variation.

Related papers

• Exploring Diffusion Transformers for Cross-Modal Augmentation in Multimodal Brain State Decoding [16.06937634283662]
  CoMA-DiT treats paired modalities as sources of mutual generative supervision rather than merely as inputs to be fused.<n>Experiments on multimodal auditory attention decoding and emotion recognition showed that CoMA-DiT consistently outperformed 20 representative baselines.
  arXiv  Detail & Related papers  (2026-09-10T10:20:10Z)
• PIDNet: Progressive Implicit Decouple Network for Multimodal Action Quality Assessment [14.92667427129547]
  Action quality assessment (AQA) aims to automatically quantify the execution quality of human actions in videos and is valuable for applications such as competitive sports judging.<n>We propose a progressive implicit decoupling and fusion network (PIDNet) that integrates modality-specific information, cross-modal complementary cues, and global quality semantics for accurate assessment.<n>Experiments on the Rhythmic Gymnastics and Fis-V datasets show that PIDNet achieves highly competitive score correlation with favorable error control compared with existing unimodal and multimodal methods.
  arXiv  Detail & Related papers  (2026-05-09T13:33:16Z)
• Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence [11.75125432258758]
  We introduce Adaptive Multimodal Intelligence (AMI), an end-to-end framework that jointly learns when to sense and how to infer.<n>AMI integrates three components: (1) a lightweight Agentic Modality Controller that uses differentiable Gumbel-Sigmoid gating to dynamically select active sensors based on model confidence and task relevance; (2) a Learned Sigma-Delta Sensing module that applies patch-wise Delta-Sigma operations with learnable thresholds to skip temporally redundant samples; and (3) a Foundation-backed Multimodal Prediction Model built on unimodal foundation encoders and a cross-modal transformer with temporal context.
  arXiv  Detail & Related papers  (2026-04-12T01:46:38Z)
• AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba [56.52470564147458]
  We propose textbfAlignMamba-2, an effective and efficient framework for multimodal fusion and sentiment analysis.<n>Our approach introduces a dual alignment strategy that regularizes the model using both Optimal Transport distance and Maximum Mean Discrepancy.<n>More importantly, we design a Modality-Aware Mamba layer, which employs a Mixture-of-Experts architecture with modality-specific and modality-shared experts.
  arXiv  Detail & Related papers  (2026-03-19T03:47:21Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• TokaMark: A Comprehensive Benchmark for MAST Tokamak Plasma Models [56.94569090844015]
  TokaMark is a structured benchmark to evaluate AI models on real experimental data collected from the Mega Ampere Spherical Tokamak (MAST)<n>TokaMark aims to accelerate progress in data-driven AI-based plasma modeling, contributing to the broader goal of achieving sustainable and stable fusion energy.
  arXiv  Detail & Related papers  (2026-02-05T16:49:44Z)
• FAIM: Frequency-Aware Interactive Mamba for Time Series Classification [87.84511960413715]
  Time series classification (TSC) is crucial in numerous real-world applications, such as environmental monitoring, medical diagnosis, and posture recognition.<n>We propose FAIM, a lightweight Frequency-Aware Interactive Mamba model.<n>We show that FAIM consistently outperforms existing state-of-the-art (SOTA) methods, achieving a superior trade-off between accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-11-26T08:36:33Z)
• MambaITD: An Efficient Cross-Modal Mamba Network for Insider Threat Detection [9.049925971684837]
  This paper proposes a new insider threat detection framework MambaITD based on the Mamba state space model and cross-modal adaptive fusion.<n>Compared with traditional methods, MambaITD shows significant advantages in modeling efficiency and feature fusion capabilities.
  arXiv  Detail & Related papers  (2025-08-06T18:45:00Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis [96.46952672172021]
  Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations. Model takes two bimodal pairs as input due to known information imbalance among modalities.
  arXiv  Detail & Related papers  (2021-07-28T23:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.