Wave-U-Mamba: An End-To-End Framework For High-Quality And Efficient Speech Super Resolution

• URL: http://arxiv.org/abs/2409.09337v2
• Date: Tue, 17 Sep 2024 17:33:57 GMT
• Title: Wave-U-Mamba: An End-To-End Framework For High-Quality And Efficient Speech Super Resolution
• Authors: Yongjoon Lee, Chanwoo Kim,
• Abstract summary: Speech Super-Resolution (SSR) is a task of enhancing low-resolution speech signals by restoring missing high-frequency components. Conventional approaches typically reconstruct log-mel features, followed by a vocoder that generates high-resolution speech in the waveform domain. We propose a method, referred to as Wave-U-Mamba, that directly performs SSR in time domain.
• Score: 4.495657539150699
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speech Super-Resolution (SSR) is a task of enhancing low-resolution speech signals by restoring missing high-frequency components. Conventional approaches typically reconstruct log-mel features, followed by a vocoder that generates high-resolution speech in the waveform domain. However, as log-mel features lack phase information, this can result in performance degradation during the reconstruction phase. Motivated by recent advances with Selective State Spaces Models (SSMs), we propose a method, referred to as Wave-U-Mamba that directly performs SSR in time domain. In our comparative study, including models such as WSRGlow, NU-Wave 2, and AudioSR, Wave-U-Mamba demonstrates superior performance, achieving the lowest Log-Spectral Distance (LSD) across various low-resolution sampling rates, ranging from 8 kHz to 24 kHz. Additionally, subjective human evaluations, scored using Mean Opinion Score (MOS) reveal that our method produces SSR with natural and human-like quality. Furthermore, Wave-U-Mamba achieves these results while generating high-resolution speech over nine times faster than baseline models on a single A100 GPU, with parameter sizes less than 2% of those in the baseline models.

Related papers

• Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization [37.35829410807451]
  This paper introduces PeriodWave-Turbo, a high-fidelity and high-efficient waveform generation model via adversarial flow matching optimization. It only requires 1,000 steps of fine-tuning to achieve state-of-the-art performance across various objective metrics. By scaling up the backbone of PeriodWave from 29M to 70M parameters for improved generalization, PeriodWave-Turbo achieves unprecedented performance.
  arXiv  Detail & Related papers  (2024-08-15T08:34:00Z)
• Frequency-Assisted Mamba for Remote Sensing Image Super-Resolution [49.902047563260496]
  We develop the first attempt to integrate the Vision State Space Model (Mamba) for remote sensing image (RSI) super-resolution. To achieve better SR reconstruction, building upon Mamba, we devise a Frequency-assisted Mamba framework, dubbed FMSR. Our FMSR features a multi-level fusion architecture equipped with the Frequency Selection Module (FSM), Vision State Space Module (VSSM), and Hybrid Gate Module (HGM)
  arXiv  Detail & Related papers  (2024-05-08T11:09:24Z)
• RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction [12.64898580131053]
  We introduce RFWave, a cutting-edge multi-band Rectified Flow approach to reconstruct high-fidelity audio waveforms from Mel-spectrograms or discrete acoustic tokens. RFWave uniquely generates complex spectrograms and operates at the frame level, processing all subbands simultaneously to boost efficiency. Our empirical evaluations show that RFWave not only provides outstanding reconstruction quality but also offers vastly superior computational efficiency, enabling audio generation at speeds up to 160 times faster than real-time on a GPU.
  arXiv  Detail & Related papers  (2024-03-08T03:16:47Z)
• mdctGAN: Taming transformer-based GAN for speech super-resolution with Modified DCT spectra [4.721572768262729]
  Speech super-resolution (SSR) aims to recover a high resolution (HR) speech from its corresponding low resolution (LR) counterpart. Recent SSR methods focus more on the reconstruction of the magnitude spectrogram, ignoring the importance of phase reconstruction. We propose mdctGAN, a novel SSR framework based on modified discrete cosine transform (MDCT)
  arXiv  Detail & Related papers  (2023-05-18T16:49:46Z)
• Synthetic Wave-Geometric Impulse Responses for Improved Speech Dereverberation [69.1351513309953]
  We show that accurately simulating the low-frequency components of Room Impulse Responses (RIRs) is important to achieving good dereverberation. We demonstrate that speech dereverberation models trained on hybrid synthetic RIRs outperform models trained on RIRs generated by prior geometric ray tracing methods.
  arXiv  Detail & Related papers  (2022-12-10T20:15:23Z)
• Neural Vocoder is All You Need for Speech Super-resolution [56.84715616516612]
  Speech super-resolution (SR) is a task to increase speech sampling rate by generating high-frequency components. Existing speech SR methods are trained in constrained experimental settings, such as a fixed upsampling ratio. We propose a neural vocoder based speech super-resolution method (NVSR) that can handle a variety of input resolution and upsampling ratios.
  arXiv  Detail & Related papers  (2022-03-28T17:51:00Z)
• A Study on Speech Enhancement Based on Diffusion Probabilistic Model [63.38586161802788]
  We propose a diffusion probabilistic model-based speech enhancement model (DiffuSE) model that aims to recover clean speech signals from noisy signals. The experimental results show that DiffuSE yields performance that is comparable to related audio generative models on the standardized Voice Bank corpus task.
  arXiv  Detail & Related papers  (2021-07-25T19:23:18Z)
• Raw Waveform Encoder with Multi-Scale Globally Attentive Locally Recurrent Networks for End-to-End Speech Recognition [45.858039215825656]
  We propose a new encoder that adopts globally attentive locally recurrent (GALR) networks and directly takes raw waveform as input. Experiments are conducted on a benchmark dataset AISHELL-2 and two large-scale Mandarin speech corpus of 5,000 hours and 21,000 hours.
  arXiv  Detail & Related papers  (2021-06-08T12:12:33Z)
• Real Time Speech Enhancement in the Waveform Domain [99.02180506016721]
  We present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is capable of removing various kinds of background noise including stationary and non-stationary noises.
  arXiv  Detail & Related papers  (2020-06-23T09:19:13Z)
• Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation [66.46123655365113]
  Target speech separation refers to extracting the target speaker's speech from mixed signals. Two main challenges are the complex acoustic environment and the real-time processing requirement. We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
  arXiv  Detail & Related papers  (2020-01-02T11:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.