Magnitude-Phase Dual-Path Speech Enhancement Network based on Self-Supervised Embedding and Perceptual Contrast Stretch Boosting

• URL: http://arxiv.org/abs/2503.21571v1
• Date: Thu, 27 Mar 2025 14:52:06 GMT
• Title: Magnitude-Phase Dual-Path Speech Enhancement Network based on Self-Supervised Embedding and Perceptual Contrast Stretch Boosting
• Authors: Alimjan Mattursun, Liejun Wang, Yinfeng Yu, Chunyang Ma,
• Abstract summary: BSP-MPNet is a dual-path framework that combines self-supervised features with magnitude-phase information for speech enhancement.<n>We evaluate BSP-MPNet on the VoiceBank+DEMAND and WHAMR! datasets.
• Score: 6.15602203132432
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Speech self-supervised learning (SSL) has made great progress in various speech processing tasks, but there is still room for improvement in speech enhancement (SE). This paper presents BSP-MPNet, a dual-path framework that combines self-supervised features with magnitude-phase information for SE. The approach starts by applying the perceptual contrast stretching (PCS) algorithm to enhance the magnitude-phase spectrum. A magnitude-phase 2D coarse (MP-2DC) encoder then extracts coarse features from the enhanced spectrum. Next, a feature-separating self-supervised learning (FS-SSL) model generates self-supervised embeddings for the magnitude and phase components separately. These embeddings are fused to create cross-domain feature representations. Finally, two parallel RNN-enhanced multi-attention (REMA) mask decoders refine the features, apply them to the mask, and reconstruct the speech signal. We evaluate BSP-MPNet on the VoiceBank+DEMAND and WHAMR! datasets. Experimental results show that BSP-MPNet outperforms existing methods under various noise conditions, providing new directions for self-supervised speech enhancement research. The implementation of the BSP-MPNet code is available online\footnote[2]{https://github.com/AlimMat/BSP-MPNet. \label{s1}}

Related papers

• SigWavNet: Learning Multiresolution Signal Wavelet Network for Speech Emotion Recognition [17.568724398229232]
  Speech emotion recognition (SER) plays an important role in emotional states from deciphering speech signals.<n>This paper introduces a new end-to-end (E2E) deep learning multi-resolution framework for SER.<n>It exploits the capabilities of wavelets for effective localization in both time and frequency domains.
  arXiv  Detail & Related papers  (2025-02-01T04:18:06Z)
• SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition [71.90536979421093]
  We propose a Split-and-Synthesize Prompting with Gated Alignments (SSPA) framework to amplify the potential of Vision-Language Models (VLMs) We develop an in-context learning approach to associate the inherent knowledge from LLMs. Then we propose a novel Split-and-Synthesize Prompting (SSP) strategy to first model the generic knowledge and downstream label semantics individually.
  arXiv  Detail & Related papers  (2024-07-30T15:58:25Z)
• Fill in the Gap! Combining Self-supervised Representation Learning with Neural Audio Synthesis for Speech Inpainting [14.402357651227003]
  We investigate the use of a speech SSL model for speech inpainting, that is reconstructing a missing portion of a speech signal from its surrounding context. To that purpose, we combine an SSL encoder, namely HuBERT, with a neural vocoder, namely HiFiGAN, playing the role of a decoder.
  arXiv  Detail & Related papers  (2024-05-30T14:41:39Z)
• Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder [58.523884148942166]
  We propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers.
  arXiv  Detail & Related papers  (2023-08-14T08:19:24Z)
• PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos [58.18707835387484]
  We propose a contrastive mask prediction framework for self-supervised learning on point cloud videos. PointCMP employs a two-branch structure to achieve simultaneous learning of both local and globaltemporal information. Our framework achieves the state-of-the-art performance on benchmark datasets and outperforms existing full-supervised counterparts.
  arXiv  Detail & Related papers  (2023-05-06T15:47:48Z)
• PARFormer: Transformer-based Multi-Task Network for Pedestrian Attribute Recognition [23.814762073093153]
  We propose a pure transformer-based multi-task PAR network named PARFormer, which includes four modules. In the feature extraction module, we build a strong baseline for feature extraction, which achieves competitive results on several PAR benchmarks. In the viewpoint perception module, we explore the impact of viewpoints on pedestrian attributes, and propose a multi-view contrastive loss. In the attribute recognition module, we alleviate the negative-positive imbalance problem to generate the attribute predictions.
  arXiv  Detail & Related papers  (2023-04-14T16:27:56Z)
• Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification [55.624946113550195]
  This paper proposes a cross-modal speech co-learning paradigm. Two cross-modal boosters are introduced based on an audio-visual pseudo-siamese structure to learn the modality-transformed correlation. Experimental results on the LRSLip3, GridLip, LomGridLip, and VoxLip datasets demonstrate that our proposed method achieves 60% and 20% average relative performance improvement.
  arXiv  Detail & Related papers  (2023-02-22T10:06:37Z)
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis [52.41439725865149]
  Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. We propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis.
  arXiv  Detail & Related papers  (2022-12-02T11:40:40Z)
• Parallel Gated Neural Network With Attention Mechanism For Speech Enhancement [0.0]
  This paper proposes a novel monaural speech enhancement system, consisting of a Feature Extraction Block (FEB), a Compensation Enhancement Block (ComEB) and a Mask Block (MB) Experiments are conducted on the Librispeech dataset and results show that the proposed model obtains better performance than recent models in terms of ESTOI and PESQ scores.
  arXiv  Detail & Related papers  (2022-10-26T06:42:19Z)
• Dual-Path Style Learning for End-to-End Noise-Robust Speech Recognition [26.77806246793544]
  Speech enhancement (SE) is introduced as front-end to reduce noise for ASR, but it also suppresses some important speech information. We propose a dual-path style learning approach for end-to-end noise-robust speech recognition (DPSL-ASR) Experiments show that the proposed approach achieves relative word error rate (WER) reductions of 10.6% and 8.6% over the best IFF-Net baseline.
  arXiv  Detail & Related papers  (2022-03-28T15:21:57Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• Multi-level Second-order Few-shot Learning [111.0648869396828]
  We propose a Multi-level Second-order (MlSo) few-shot learning network for supervised or unsupervised few-shot image classification and few-shot action recognition. We leverage so-called power-normalized second-order base learner streams combined with features that express multiple levels of visual abstraction. We demonstrate respectable results on standard datasets such as Omniglot, mini-ImageNet, tiered-ImageNet, Open MIC, fine-grained datasets such as CUB Birds, Stanford Dogs and Cars, and action recognition datasets such as HMDB51, UCF101, and mini-MIT.
  arXiv  Detail & Related papers  (2022-01-15T19:49:00Z)
• Video Saliency Prediction Using Enhanced Spatiotemporal Alignment Network [35.932447204088845]
  We develop an effective feature alignment network tailored to video saliency prediction (V) The network learns to align the features of the neighboring frames to the reference one in a coarse-to-fine manner. The proposed model is trained end-to-end without any post processing.
  arXiv  Detail & Related papers  (2020-01-02T02:05:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.