High-Fidelity Audio Compression with Improved RVQGAN
- URL: http://arxiv.org/abs/2306.06546v2
- Date: Thu, 26 Oct 2023 22:17:49 GMT
- Title: High-Fidelity Audio Compression with Improved RVQGAN
- Authors: Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, Kundan
Kumar
- Abstract summary: We introduce a high-fidelity universal neural audio compression algorithm that achieves 90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth.
We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio.
- Score: 49.7859037103693
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Language models have been successfully used to model natural signals, such as
images, speech, and music. A key component of these models is a high quality
neural compression model that can compress high-dimensional natural signals
into lower dimensional discrete tokens. To that end, we introduce a
high-fidelity universal neural audio compression algorithm that achieves ~90x
compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve
this by combining advances in high-fidelity audio generation with better vector
quantization techniques from the image domain, along with improved adversarial
and reconstruction losses. We compress all domains (speech, environment, music,
etc.) with a single universal model, making it widely applicable to generative
modeling of all audio. We compare with competing audio compression algorithms,
and find our method outperforms them significantly. We provide thorough
ablations for every design choice, as well as open-source code and trained
model weights. We hope our work can lay the foundation for the next generation
of high-fidelity audio modeling.
Related papers
- From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion [84.138804145918]
Deep generative models can generate high-fidelity audio conditioned on various types of representations.
These models are prone to generate audible artifacts when the conditioning is flawed or imperfect.
We propose a high-fidelity multi-band diffusion-based framework that generates any type of audio modality from low-bitrate discrete representations.
arXiv Detail & Related papers (2023-08-02T22:14:29Z) - Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion
Models [65.18102159618631]
multimodal generative modeling has created milestones in text-to-image and text-to-video generation.
Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data.
We propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps.
arXiv Detail & Related papers (2023-01-30T04:44:34Z) - High Fidelity Neural Audio Compression [92.4812002532009]
We introduce a state-of-the-art real-time, high-fidelity, audio leveraging neural networks.
It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion.
We simplify and speed-up the training by using a single multiscale spectrogram adversary.
arXiv Detail & Related papers (2022-10-24T17:52:02Z) - BigVGAN: A Universal Neural Vocoder with Large-Scale Training [49.16254684584935]
We present BigVGAN, a universal vocoder that generalizes well under various unseen conditions in zero-shot setting.
We introduce periodic nonlinearities and anti-aliased representation into the generator, which brings the desired inductive bias for waveform.
We train our GAN vocoder at the largest scale up to 112M parameters, which is unprecedented in the literature.
arXiv Detail & Related papers (2022-06-09T17:56:10Z) - RAVE: A variational autoencoder for fast and high-quality neural audio
synthesis [2.28438857884398]
We introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis.
We show that our model is the first able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU.
arXiv Detail & Related papers (2021-11-09T09:07:30Z) - SoundStream: An End-to-End Neural Audio Codec [78.94923131038682]
We present SoundStream, a novel neural audio system that can efficiently compress speech, music and general audio.
SoundStream relies on a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end.
We are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency.
arXiv Detail & Related papers (2021-07-07T15:45:42Z) - MP3net: coherent, minute-long music generation from raw audio with a
simple convolutional GAN [0.0]
We present a deep convolutional GAN which produces high-quality audio samples with long-range coherence.
We leverage the auditory masking and psychoacoustic perception limit of the human ear to widen the true distribution.
We use MP3net to create 95s stereo tracks with a 22kHz sample rate after training for 250h on a single Cloud TPUv2.
arXiv Detail & Related papers (2021-01-12T22:37:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.