Neural Waveshaping Synthesis

• URL: http://arxiv.org/abs/2107.05050v1
• Date: Sun, 11 Jul 2021 13:50:59 GMT
• Title: Neural Waveshaping Synthesis
• Authors: Ben Hayes, Charalampos Saitis, Gy\"orgy Fazekas
• Abstract summary: We present a novel, lightweight, fully causal approach to neural audio synthesis. The Neural Waveshaping Unit (NEWT) operates directly in the waveform domain. It produces complex timbral evolutions by simple affine transformations of its input and output signals.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present the Neural Waveshaping Unit (NEWT): a novel, lightweight, fully causal approach to neural audio synthesis which operates directly in the waveform domain, with an accompanying optimisation (FastNEWT) for efficient CPU inference. The NEWT uses time-distributed multilayer perceptrons with periodic activations to implicitly learn nonlinear transfer functions that encode the characteristics of a target timbre. Once trained, a NEWT can produce complex timbral evolutions by simple affine transformations of its input and output signals. We paired the NEWT with a differentiable noise synthesiser and reverb and found it capable of generating realistic musical instrument performances with only 260k total model parameters, conditioned on F0 and loudness features. We compared our method to state-of-the-art benchmarks with a multi-stimulus listening test and the Fr\'echet Audio Distance and found it performed competitively across the tested timbral domains. Our method significantly outperformed the benchmarks in terms of generation speed, and achieved real-time performance on a consumer CPU, both with and without FastNEWT, suggesting it is a viable basis for future creative sound design tools.

Related papers

• Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion [85.54515118077825]
  This paper proposes a linear diffusion model (LinDiff) based on an ordinary differential equation to simultaneously reach fast inference and high sample quality. To reduce computational complexity, LinDiff employs a patch-based processing approach that partitions the input signal into small patches. Our model can synthesize speech of a quality comparable to that of autoregressive models with faster synthesis speed.
  arXiv  Detail & Related papers  (2023-06-09T07:02:43Z)
• An investigation of the reconstruction capacity of stacked convolutional autoencoders for log-mel-spectrograms [2.3204178451683264]
  In audio processing applications, the generation of expressive sounds based on high-level representations demonstrates a high demand. Modern algorithms, such as neural networks, have inspired the development of expressive synthesizers based on musical instrument compression. This study investigates the use of stacked convolutional autoencoders for the compression of time-frequency audio representations for a variety of instruments for a single pitch.
  arXiv  Detail & Related papers  (2023-01-18T17:19:04Z)
• Synthesizer Preset Interpolation using Transformer Auto-Encoders [4.213427823201119]
  We introduce a bimodal auto-encoder neural network, which simultaneously processes presets using multi-head attention blocks, and audio using convolutions. This model has been tested on a popular frequency modulation synthesizer with more than one hundred parameters. After training, the proposed model can be integrated into commercial synthesizers for live or sound design tasks.
  arXiv  Detail & Related papers  (2022-10-27T15:20:18Z)
• FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis [90.3069686272524]
  This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies. Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms.
  arXiv  Detail & Related papers  (2022-04-21T07:49:09Z)
• NeuralDPS: Neural Deterministic Plus Stochastic Model with Multiband Excitation for Noise-Controllable Waveform Generation [67.96138567288197]
  We propose a novel neural vocoder named NeuralDPS which can retain high speech quality and acquire high synthesis efficiency and noise controllability. It generates waveforms at least 280 times faster than the WaveNet vocoder. It is also 28% faster than WaveGAN's synthesis efficiency on a single core.
  arXiv  Detail & Related papers  (2022-03-05T08:15:29Z)
• RAVE: A variational autoencoder for fast and high-quality neural audio synthesis [2.28438857884398]
  We introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis. We show that our model is the first able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU.
  arXiv  Detail & Related papers  (2021-11-09T09:07:30Z)
• Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning [42.95813372611093]
  We propose to generate sounds conditioned on sound classes via neural discrete time-frequency representation learning. This offers an advantage in modelling long-range dependencies and retaining local fine-grained structure within a sound clip.
  arXiv  Detail & Related papers  (2021-07-21T10:31:28Z)
• Deep Convolutional and Recurrent Networks for Polyphonic Instrument Classification from Monophonic Raw Audio Waveforms [30.3491261167433]
  Sound Event Detection and Audio Classification tasks are traditionally addressed through time-frequency representations of audio signals such as spectrograms. Deep neural networks as efficient feature extractors has enabled the direct use of audio signals for classification purposes. We attempt to recognize musical instruments in polyphonic audio by only feeding their raw waveforms into deep learning models.
  arXiv  Detail & Related papers  (2021-02-13T13:44:46Z)
• Fast accuracy estimation of deep learning based multi-class musical source separation [79.10962538141445]
  We propose a method to evaluate the separability of instruments in any dataset without training and tuning a neural network. Based on the oracle principle with an ideal ratio mask, our approach is an excellent proxy to estimate the separation performances of state-of-the-art deep learning approaches.
  arXiv  Detail & Related papers  (2020-10-19T13:05:08Z)
• Real Time Speech Enhancement in the Waveform Domain [99.02180506016721]
  We present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is capable of removing various kinds of background noise including stationary and non-stationary noises.
  arXiv  Detail & Related papers  (2020-06-23T09:19:13Z)
• VaPar Synth -- A Variational Parametric Model for Audio Synthesis [78.3405844354125]
  We present VaPar Synth - a Variational Parametric Synthesizer which utilizes a conditional variational autoencoder (CVAE) trained on a suitable parametric representation. We demonstrate our proposed model's capabilities via the reconstruction and generation of instrumental tones with flexible control over their pitch.
  arXiv  Detail & Related papers  (2020-03-30T16:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.