Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization

• URL: http://arxiv.org/abs/2408.08019v1
• Date: Thu, 15 Aug 2024 08:34:00 GMT
• Title: Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization
• Authors: Sang-Hoon Lee, Ha-Yeong Choi, Seong-Whan Lee,
• Abstract summary: This paper introduces PeriodWave-Turbo, a high-fidelity and high-efficient waveform generation model via adversarial flow matching optimization. It only requires 1,000 steps of fine-tuning to achieve state-of-the-art performance across various objective metrics. By scaling up the backbone of PeriodWave from 29M to 70M parameters for improved generalization, PeriodWave-Turbo achieves unprecedented performance.
• Score: 37.35829410807451
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper introduces PeriodWave-Turbo, a high-fidelity and high-efficient waveform generation model via adversarial flow matching optimization. Recently, conditional flow matching (CFM) generative models have been successfully adopted for waveform generation tasks, leveraging a single vector field estimation objective for training. Although these models can generate high-fidelity waveform signals, they require significantly more ODE steps compared to GAN-based models, which only need a single generation step. Additionally, the generated samples often lack high-frequency information due to noisy vector field estimation, which fails to ensure high-frequency reproduction. To address this limitation, we enhance pre-trained CFM-based generative models by incorporating a fixed-step generator modification. We utilized reconstruction losses and adversarial feedback to accelerate high-fidelity waveform generation. Through adversarial flow matching optimization, it only requires 1,000 steps of fine-tuning to achieve state-of-the-art performance across various objective metrics. Moreover, we significantly reduce inference speed from 16 steps to 2 or 4 steps. Additionally, by scaling up the backbone of PeriodWave from 29M to 70M parameters for improved generalization, PeriodWave-Turbo achieves unprecedented performance, with a perceptual evaluation of speech quality (PESQ) score of 4.454 on the LibriTTS dataset. Audio samples, source code and checkpoints will be available at https://github.com/sh-lee-prml/PeriodWave.

Related papers

• PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation [37.35829410807451]
  We propose PeriodWave, a novel universal waveform generation model. We introduce a period-aware flow matching estimator that can capture the periodic features of the waveform signal. We also propose a single period-conditional universal estimator that can feed-forward parallel by period-wise batch inference.
  arXiv  Detail & Related papers  (2024-08-14T13:36:17Z)
• RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction [12.64898580131053]
  We introduce RFWave, a cutting-edge multi-band Rectified Flow approach to reconstruct high-fidelity audio waveforms from Mel-spectrograms or discrete acoustic tokens. RFWave uniquely generates complex spectrograms and operates at the frame level, processing all subbands simultaneously to boost efficiency. Our empirical evaluations show that RFWave not only provides outstanding reconstruction quality but also offers vastly superior computational efficiency, enabling audio generation at speeds up to 160 times faster than real-time on a GPU.
  arXiv  Detail & Related papers  (2024-03-08T03:16:47Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)
• FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis [90.3069686272524]
  This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies. Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms.
  arXiv  Detail & Related papers  (2022-04-21T07:49:09Z)
• A Study on Speech Enhancement Based on Diffusion Probabilistic Model [63.38586161802788]
  We propose a diffusion probabilistic model-based speech enhancement model (DiffuSE) model that aims to recover clean speech signals from noisy signals. The experimental results show that DiffuSE yields performance that is comparable to related audio generative models on the standardized Voice Bank corpus task.
  arXiv  Detail & Related papers  (2021-07-25T19:23:18Z)
• High-Fidelity and Low-Latency Universal Neural Vocoder based on Multiband WaveRNN with Data-Driven Linear Prediction for Discrete Waveform Modeling [38.828260316517536]
  This paper presents a novel universal neural vocoder framework based on multiband WaveRNN with data-driven linear prediction for discrete waveform modeling (MWDLP) Experiments demonstrate that the proposed MWDLP framework generates high-fidelity synthetic speech for seen and unseen speakers and/or language on 300 speakers training data including clean and noisy/reverberant conditions.
  arXiv  Detail & Related papers  (2021-05-20T16:02:45Z)
• DiffWave: A Versatile Diffusion Model for Audio Synthesis [35.406438835268816]
  DiffWave is a versatile diffusion probabilistic model for conditional and unconditional waveform generation. It produces high-fidelity audios in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram. It significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task.
  arXiv  Detail & Related papers  (2020-09-21T11:20:38Z)
• WaveGrad: Estimating Gradients for Waveform Generation [55.405580817560754]
  WaveGrad is a conditional model for waveform generation which estimates gradients of the data density. It starts from a Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned on the mel-spectrogram. We find that it can generate high fidelity audio samples using as few as six iterations.
  arXiv  Detail & Related papers  (2020-09-02T17:44:10Z)
• Real Time Speech Enhancement in the Waveform Domain [99.02180506016721]
  We present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is capable of removing various kinds of background noise including stationary and non-stationary noises.
  arXiv  Detail & Related papers  (2020-06-23T09:19:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.