BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embedding

• URL: http://arxiv.org/abs/2408.06851v1
• Date: Tue, 13 Aug 2024 12:27:24 GMT
• Title: BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embedding
• Authors: Alimjan Mattursun, Liejun Wang, Yinfeng Yu,
• Abstract summary: Speech self-supervised learning (SSL) represents has achieved state-of-the-art (SOTA) performance in multiple downstream tasks. This study introduces a novel cross-domain feature fusion and multi-attention speech enhancement network, termed BSS-CFFMA. We evaluate the performance of the BSS-CFFMA model through comparative and ablation studies on the VoiceBank-DEMAND dataset.
• Score: 6.725011823614421
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Speech self-supervised learning (SSL) represents has achieved state-of-the-art (SOTA) performance in multiple downstream tasks. However, its application in speech enhancement (SE) tasks remains immature, offering opportunities for improvement. In this study, we introduce a novel cross-domain feature fusion and multi-attention speech enhancement network, termed BSS-CFFMA, which leverages self-supervised embeddings. BSS-CFFMA comprises a multi-scale cross-domain feature fusion (MSCFF) block and a residual hybrid multi-attention (RHMA) block. The MSCFF block effectively integrates cross-domain features, facilitating the extraction of rich acoustic information. The RHMA block, serving as the primary enhancement module, utilizes three distinct attention modules to capture diverse attention representations and estimate high-quality speech signals. We evaluate the performance of the BSS-CFFMA model through comparative and ablation studies on the VoiceBank-DEMAND dataset, achieving SOTA results. Furthermore, we select three types of data from the WHAMR! dataset, a collection specifically designed for speech enhancement tasks, to assess the capabilities of BSS-CFFMA in tasks such as denoising only, dereverberation only, and simultaneous denoising and dereverberation. This study marks the first attempt to explore the effectiveness of self-supervised embedding-based speech enhancement methods in complex tasks encompassing dereverberation and simultaneous denoising and dereverberation. The demo implementation of BSS-CFFMA is available online\footnote[2]{https://github.com/AlimMat/BSS-CFFMA. \label{s1}}.

Related papers

• FAIM: Frequency-Aware Interactive Mamba for Time Series Classification [87.84511960413715]
  Time series classification (TSC) is crucial in numerous real-world applications, such as environmental monitoring, medical diagnosis, and posture recognition.<n>We propose FAIM, a lightweight Frequency-Aware Interactive Mamba model.<n>We show that FAIM consistently outperforms existing state-of-the-art (SOTA) methods, achieving a superior trade-off between accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-11-26T08:36:33Z)
• Joint Learning using Mixture-of-Expert-Based Representation for Enhanced Speech Generation and Robust Emotion Recognition [54.44798086835314]
  Speech emotion recognition (SER) plays a critical role in building emotion-aware speech systems, but its performance degrades significantly under noisy conditions.<n>We propose the Sparse Mixture-of-Experts Representation Integration Technique (Sparse MERIT), a flexible MTL framework that applies frame-wise expert routing over self-supervised speech representations.<n> Experiments on the MSP-Podcast corpus show that Sparse MERIT consistently outperforms baseline models on both SER and SE tasks.
  arXiv  Detail & Related papers  (2025-09-10T10:18:56Z)
• Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder [53.00939565103065]
  We present a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) tasks.<n>We leverage the hidden representations from multiple layers of UME as a residual weighted-sum encoding (RWSE) to effectively use information from different semantic levels.<n>This joint training approach captures the inherent interdependencies among the tasks, enhancing overall performance on overlapping speech data.
  arXiv  Detail & Related papers  (2025-08-28T06:50:57Z)
• Magnitude-Phase Dual-Path Speech Enhancement Network based on Self-Supervised Embedding and Perceptual Contrast Stretch Boosting [6.15602203132432]
  BSP-MPNet is a dual-path framework that combines self-supervised features with magnitude-phase information for speech enhancement.<n>We evaluate BSP-MPNet on the VoiceBank+DEMAND and WHAMR! datasets.
  arXiv  Detail & Related papers  (2025-03-27T14:52:06Z)
• MExD: An Expert-Infused Diffusion Model for Whole-Slide Image Classification [46.89908887119571]
  Whole Slide Image (WSI) classification poses unique challenges due to the vast image size and numerous non-informative regions. We propose MExD, an Expert-Infused Diffusion Model that combines the strengths of a Mixture-of-Experts (MoE) mechanism with a diffusion model for enhanced classification.
  arXiv  Detail & Related papers  (2025-03-16T08:04:17Z)
• Disentangling CLIP Features for Enhanced Localized Understanding [58.73850193789384]
  We propose Unmix-CLIP, a novel framework designed to reduce mutual feature information (MFI) and improve feature disentanglement. For the COCO- 14 dataset, Unmix-CLIP reduces feature similarity by 24.9%.
  arXiv  Detail & Related papers  (2025-02-05T08:20:31Z)
• Generalized Uncertainty-Based Evidential Fusion with Hybrid Multi-Head Attention for Weak-Supervised Temporal Action Localization [28.005080560540133]
  Weakly supervised temporal action localization (WS-TAL) is a task of targeting at localizing complete action instances and categorizing them with video-level labels. Action-background ambiguity, primarily caused by background noise resulting from aggregation and intra-action variation, is a significant challenge for existing WS-TAL methods. We introduce a hybrid multi-head attention (HMHA) module and generalized uncertainty-based evidential fusion (GUEF) module to address the problem.
  arXiv  Detail & Related papers  (2024-12-27T03:04:57Z)
• CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation [18.276988929148143]
  We explore the potential of large-scale noisily labeled data to enhance feature learning by pretraining semantic segmentation models. Unlike conventional pretraining approaches, CromSS exploits massive amounts of noisy and easy-to-come-by labels for improved feature learning.
  arXiv  Detail & Related papers  (2024-05-02T11:58:06Z)
• SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representations [23.56580783289533]
  This work presents a cost-effective SSFT method named Self-supervised Correspondence (SCORE) fine-tuning to adapt the SSL speech representations for content-related tasks. SCORE outperforms the vanilla HuBERT on SUPERB benchmark with only a few hours of fine-tuning ( 5 hrs) on a single GPU for automatic speech recognition, phoneme recognition, and query-by-example tasks.
  arXiv  Detail & Related papers  (2024-03-10T16:57:51Z)
• DiffVein: A Unified Diffusion Network for Finger Vein Segmentation and Authentication [50.017055360261665]
  We introduce DiffVein, a unified diffusion model-based framework which simultaneously addresses vein segmentation and authentication tasks. For better feature interaction between these two branches, we introduce two specialized modules. In this way, our framework allows for a dynamic interplay between diffusion and segmentation embeddings.
  arXiv  Detail & Related papers  (2024-02-03T06:49:42Z)
• MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition [62.89464258519723]
  We propose a multi-layer cross-attention fusion based AVSR approach that promotes representation of each modality by fusing them at different levels of audio/visual encoders. Our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset.
  arXiv  Detail & Related papers  (2024-01-07T08:59:32Z)
• IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation [36.935137240527204]
  We propose a novel model called Intra- and Inter-Attention Network (IIANet), which leverages the attention mechanism for efficient audio-visual feature fusion. IIANet consists of two types of attention blocks: intra-attention (IntraA) and inter-attention (InterA) blocks, where the InterA blocks are distributed at the top, middle and bottom of IIANet. Comprehensive experiments on three standard audio-visual separation benchmarks (LRS2, LRS3, and VoxCeleb2) demonstrate the effectiveness of IIANet, outperforming previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-16T04:31:33Z)
• Dense Affinity Matching for Few-Shot Segmentation [83.65203917246745]
  Few-Shot (FSS) aims to segment the novel class images with a few samples. We propose a dense affinity matching framework to exploit the support-query interaction. We show that our framework performs very competitively under different settings with only 0.68M parameters.
  arXiv  Detail & Related papers  (2023-07-17T12:27:15Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• Parallel Gated Neural Network With Attention Mechanism For Speech Enhancement [0.0]
  This paper proposes a novel monaural speech enhancement system, consisting of a Feature Extraction Block (FEB), a Compensation Enhancement Block (ComEB) and a Mask Block (MB) Experiments are conducted on the Librispeech dataset and results show that the proposed model obtains better performance than recent models in terms of ESTOI and PESQ scores.
  arXiv  Detail & Related papers  (2022-10-26T06:42:19Z)
• End-to-End Active Speaker Detection [58.7097258722291]
  We propose an end-to-end training network where feature learning and contextual predictions are jointly learned. We also introduce intertemporal graph neural network (iGNN) blocks, which split the message passing according to the main sources of context in the ASD problem. Experiments show that the aggregated features from the iGNN blocks are more suitable for ASD, resulting in state-of-the art performance.
  arXiv  Detail & Related papers  (2022-03-27T08:55:28Z)
• A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition [7.80238628278552]
  We propose a novel cross-modal fusion network based on self-attention and residual structure (CFN-SR) for multimodal emotion recognition. To verify the effectiveness of the proposed method, we conduct experiments on the RAVDESS dataset. The experimental results show that the proposed CFN-SR achieves the state-of-the-art and obtains 75.76% accuracy with 26.30M parameters.
  arXiv  Detail & Related papers  (2021-11-03T12:24:03Z)
• Multi-Granularity Reference-Aided Attentive Feature Aggregation for Video-based Person Re-identification [98.7585431239291]
  Video-based person re-identification aims at matching the same person across video clips. In this paper, we propose an attentive feature aggregation module, namely Multi-Granularity Reference-Attentive Feature aggregation module MG-RAFA. Our framework achieves the state-of-the-art ablation performance on three benchmark datasets.
  arXiv  Detail & Related papers  (2020-03-27T03:49:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.