MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement

• URL: http://arxiv.org/abs/2507.00966v1
• Date: Tue, 01 Jul 2025 17:16:05 GMT
• Title: MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement
• Authors: Nikolai Lund Kühne, Jesper Jensen, Jan Østergaard, Zheng-Hua Tan,
• Abstract summary: We propose a novel hybrid architecture, MambAttention, which combines Mamba and shared time- and frequency-multi-head attention modules.<n>Our proposed MambAttention model significantly outperforms existing state-of-the-art LSTM-, xLSTM-, Mamba-, and Conformer-based systems.
• Score: 19.76560732937885
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: With the advent of new sequence models like Mamba and xLSTM, several studies have shown that these models match or outperform state-of-the-art models in single-channel speech enhancement, automatic speech recognition, and self-supervised audio representation learning. However, prior research has demonstrated that sequence models like LSTM and Mamba tend to overfit to the training set. To address this issue, previous works have shown that adding self-attention to LSTMs substantially improves generalization performance for single-channel speech enhancement. Nevertheless, neither the concept of hybrid Mamba and time-frequency attention models nor their generalization performance have been explored for speech enhancement. In this paper, we propose a novel hybrid architecture, MambAttention, which combines Mamba and shared time- and frequency-multi-head attention modules for generalizable single-channel speech enhancement. To train our model, we introduce VoiceBank+Demand Extended (VB-DemandEx), a dataset inspired by VoiceBank+Demand but with more challenging noise types and lower signal-to-noise ratios. Trained on VB-DemandEx, our proposed MambAttention model significantly outperforms existing state-of-the-art LSTM-, xLSTM-, Mamba-, and Conformer-based systems of similar complexity across all reported metrics on two out-of-domain datasets: DNS 2020 and EARS-WHAM_v2, while matching their performance on the in-domain dataset VB-DemandEx. Ablation studies highlight the role of weight sharing between the time- and frequency-multi-head attention modules for generalization performance. Finally, we explore integrating the shared time- and frequency-multi-head attention modules with LSTM and xLSTM, which yields a notable performance improvement on the out-of-domain datasets. However, our MambAttention model remains superior on both out-of-domain datasets across all reported evaluation metrics.

Related papers

• Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection [88.47928738482719]
  Linear State Space Models (SSMs) offer remarkable performance gains in sequence modeling.<n>Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations.<n>We introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts.
  arXiv  Detail & Related papers  (2025-06-22T19:26:55Z)
• An Exploration of Mamba for Speech Self-Supervised Models [48.01992287080999]
  We explore Mamba-based HuBERT models as alternatives to Transformer-based SSL architectures.<n>HuBERT models enable fine-tuning on long-context ASR with significantly lower compute.<n>These findings highlight Mamba-based SSL as a promising and complementary direction for long-sequence modeling, real-time speech modeling, and speech unit extraction.
  arXiv  Detail & Related papers  (2025-06-14T19:00:44Z)
• M2Rec: Multi-scale Mamba for Efficient Sequential Recommendation [35.508076394809784]
  model is a novel sequential recommendation framework that integrates multi-scale Mamba with Fourier analysis, Large Language Models, and adaptive gating.<n>Experiments demonstrate that model achieves state-of-the-art performance, improving Hit Rate@10 by 3.2% over existing Mamba-based models.
  arXiv  Detail & Related papers  (2025-05-07T14:14:29Z)
• RD-UIE: Relation-Driven State Space Modeling for Underwater Image Enhancement [59.364418120895]
  Underwater image enhancement (UIE) is a critical preprocessing step for marine vision applications.<n>We develop a novel relation-driven Mamba framework for effective UIE (RD-UIE)<n>Experiments on underwater enhancement benchmarks demonstrate RD-UIE outperforms the state-of-the-art approach WMamba.
  arXiv  Detail & Related papers  (2025-05-02T12:21:44Z)
• A Deep Learning Framework for Sequence Mining with Bidirectional LSTM and Multi-Scale Attention [11.999319439383918]
  This paper addresses the challenges of mining latent patterns and modeling contextual dependencies in complex sequence data.<n>A sequence pattern mining algorithm is proposed by integrating Bidirectional Long Short-Term Memory (BiLSTM) with a multi-scale attention mechanism.<n>BiLSTM captures both forward and backward dependencies in sequences, enhancing the model's ability to perceive global contextual structures.
  arXiv  Detail & Related papers  (2025-04-21T16:53:02Z)
• xLSTM-SENet: xLSTM for Single-Channel Speech Enhancement [19.76560732937885]
  This paper introduces xLSTM-SENet, the first xLSTM-based single-channel speech enhancement system.<n>Our best xLSTM-based model, xLSTM-SENet2, outperforms state-of-the-art Mamba- and Conformer-based systems of similar complexity on the Voicebank+DEMAND dataset.
  arXiv  Detail & Related papers  (2025-01-10T18:10:06Z)
• Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement [54.427965535613886]
  Mamba, as a novel state-space model (SSM), has gained widespread application in natural language processing and computer vision.<n>In this work, we introduce Mamba-SEUNet, an innovative architecture that integrates Mamba with U-Net for SE tasks.
  arXiv  Detail & Related papers  (2024-12-21T13:43:51Z)
• UniRAG: Universal Retrieval Augmentation for Large Vision Language Models [76.30799731147589]
  We introduce UniRAG, a plug-and-play technique that adds relevant retrieved information to prompts as few-shot examples during inference.<n>Unlike the common belief that Retrieval Augmentation (RA) mainly improves generation or understanding of uncommon entities, our evaluation results on the MSCOCO dataset with common entities show that both proprietary models and smaller open-source models significantly enhance their generation quality.
  arXiv  Detail & Related papers  (2024-05-16T17:58:45Z)
• MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection [5.37935922811333]
  MambaMixer is a new architecture with data-dependent weights that uses a dual selection mechanism across tokens and channels. As a proof of concept, we design Vision MambaMixer (ViM2) and Time Series MambaMixer (TSM2) architectures based on the MambaMixer block.
  arXiv  Detail & Related papers  (2024-03-29T00:05:13Z)
• Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks [55.36987468073152]
  This paper proposes a novel Dual-Guided Spatial-Channel-Temporal (DG-SCT) attention mechanism. The DG-SCT module incorporates trainable cross-modal interaction layers into pre-trained audio-visual encoders. Our proposed model achieves state-of-the-art results across multiple downstream tasks, including AVE, AVVP, AVS, and AVQA.
  arXiv  Detail & Related papers  (2023-11-09T05:24:20Z)
• Attention Bottlenecks for Multimodal Fusion [90.75885715478054]
  Machine perception models are typically modality-specific and optimised for unimodal benchmarks. We introduce a novel transformer based architecture that uses fusion' for modality fusion at multiple layers. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks.
  arXiv  Detail & Related papers  (2021-06-30T22:44:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.