Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-To-End Audio Style Transfer

• URL: http://arxiv.org/abs/2208.07282v5
• Date: Mon, 8 May 2023 13:45:05 GMT
• Title: Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-To-End Audio Style Transfer
• Authors: Shahan Nercessian
• Abstract summary: We propose a differentiable WORLD synthesizer and demonstrate its use in end-to-end audio style transfer tasks. Our baseline differentiable synthesizer has no model parameters, yet it yields adequate quality synthesis. An alternative differentiable approach considers extraction of the source spectrum directly, which can improve naturalness.
• Score: 6.29475963948119
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose a differentiable WORLD synthesizer and demonstrate its use in end-to-end audio style transfer tasks such as (singing) voice conversion and the DDSP timbre transfer task. Accordingly, our baseline differentiable synthesizer has no model parameters, yet it yields adequate synthesis quality. We can extend the baseline synthesizer by appending lightweight black-box postnets which apply further processing to the baseline output in order to improve fidelity. An alternative differentiable approach considers extraction of the source excitation spectrum directly, which can improve naturalness albeit for a narrower class of style transfer applications. The acoustic feature parameterization used by our approaches has the added benefit that it naturally disentangles pitch and timbral information so that they can be modeled separately. Moreover, as there exists a robust means of estimating these acoustic features from monophonic audio sources, it allows for parameter loss terms to be added to an end-to-end objective function, which can help convergence and/or further stabilize (adversarial) training.

Related papers

• Robust AI-Synthesized Speech Detection Using Feature Decomposition Learning and Synthesizer Feature Augmentation [52.0893266767733]
  We propose a robust deepfake speech detection method that employs feature decomposition to learn synthesizer-independent content features. To enhance the model's robustness to different synthesizer characteristics, we propose a synthesizer feature augmentation strategy.
  arXiv  Detail & Related papers  (2024-11-14T03:57:21Z)
• DiffMoog: a Differentiable Modular Synthesizer for Sound Matching [48.33168531500444]
  DiffMoog is a differentiable modular synthesizer with a comprehensive set of modules typically found in commercial instruments. Being differentiable, it allows integration into neural networks, enabling automated sound matching. We introduce an open-source platform that comprises DiffMoog and an end-to-end sound matching framework.
  arXiv  Detail & Related papers  (2024-01-23T08:59:21Z)
• DDX7: Differentiable FM Synthesis of Musical Instrument Sounds [7.829520196474829]
  Differentiable Digital Signal Processing (DDSP) has enabled nuanced audio rendering by Deep Neural Networks (DNNs) We present Differentiable DX7 (DDX7), a lightweight architecture for neural FM resynthesis of musical instrument sounds.
  arXiv  Detail & Related papers  (2022-08-12T08:39:45Z)
• Differentiable Digital Signal Processing Mixture Model for Synthesis Parameter Extraction from Mixture of Harmonic Sounds [29.012177604120048]
  A differentiable digital signal processing (DDSP) autoencoder is a musical sound that combines a deep neural network (DNN) and spectral modeling synthesis. It allows us to flexibly edit sounds by changing the fundamental frequency, timbre feature, and loudness (synthesis parameters) extracted from an input sound. It is designed for a monophonic harmonic sound and cannot handle mixtures of sounds harmonic.
  arXiv  Detail & Related papers  (2022-02-01T03:38:49Z)
• Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control [47.33830090185952]
  A text-to-rapping/singing system is introduced, which can be adapted to any speaker's voice. It utilizes a Tacotron-based multispeaker acoustic model trained on read-only speech data. Results show that the proposed approach can produce high quality rapping/singing voice with increased naturalness.
  arXiv  Detail & Related papers  (2021-11-17T14:31:55Z)
• DiffSinger: Diffusion Acoustic Model for Singing Voice Synthesis [53.19363127760314]
  DiffSinger is a parameterized Markov chain which iteratively converts the noise into mel-spectrogram conditioned on the music score. The evaluations conducted on the Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work with a notable margin.
  arXiv  Detail & Related papers  (2021-05-06T05:21:42Z)
• Vector-Quantized Timbre Representation [53.828476137089325]
  This paper targets a more flexible synthesis of an individual timbre by learning an approximate decomposition of its spectral properties with a set of generative features. We introduce an auto-encoder with a discrete latent space that is disentangled from loudness in order to learn a quantized representation of a given timbre distribution. We detail results for translating audio between orchestral instruments and singing voice, as well as transfers from vocal imitations to instruments.
  arXiv  Detail & Related papers  (2020-07-13T12:35:45Z)
• VaPar Synth -- A Variational Parametric Model for Audio Synthesis [78.3405844354125]
  We present VaPar Synth - a Variational Parametric Synthesizer which utilizes a conditional variational autoencoder (CVAE) trained on a suitable parametric representation. We demonstrate our proposed model's capabilities via the reconstruction and generation of instrumental tones with flexible control over their pitch.
  arXiv  Detail & Related papers  (2020-03-30T16:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.