A2SB: Audio-to-Audio Schrodinger Bridges

• URL: http://arxiv.org/abs/2501.11311v1
• Date: Mon, 20 Jan 2025 07:28:41 GMT
• Title: A2SB: Audio-to-Audio Schrodinger Bridges
• Authors: Zhifeng Kong, Kevin J Shih, Weili Nie, Arash Vahdat, Sang-gil Lee, Joao Felipe Santos, Ante Jukic, Rafael Valle, Bryan Catanzaro,
• Abstract summary: We present an audio restoration model for high-res music at 44.1kHz.<n>Our model, Audio-to-Audio Schrodinger Bridges (A2SB), is capable of both bandwidth extension and inpainting.
• Score: 52.637226461175096
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio in the real world may be perturbed due to numerous factors, causing the audio quality to be degraded. The following work presents an audio restoration model tailored for high-res music at 44.1kHz. Our model, Audio-to-Audio Schrodinger Bridges (A2SB), is capable of both bandwidth extension (predicting high-frequency components) and inpainting (re-generating missing segments). Critically, A2SB is end-to-end without need of a vocoder to predict waveform outputs, able to restore hour-long audio inputs, and trained on permissively licensed music data. A2SB is capable of achieving state-of-the-art bandwidth extension and inpainting quality on several out-of-distribution music test sets. Our demo website is https: //research.nvidia.com/labs/adlr/A2SB/.

Related papers

• Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities [72.91296768332163]
  We introduce Audio Flamingo 2 (AF2), an Audio-Language Model, and LongAudio, a dataset for training ALMs on long audio captioning and question-answering tasks. AF2 achieves state-of-the-art performance with only a 3B parameter small language model, surpassing large open-source and proprietary models across over 20 benchmarks. For the first time, we extend audio understanding to long audio segments (30 secs to 5 mins) and propose LongAudio, a large and novel dataset for training ALMs on long audio captioning and question-answering tasks.
  arXiv  Detail & Related papers  (2025-03-06T00:10:26Z)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [51.70360630470263]
  Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video.<n>We propose Frieren, a V2A model based on rectified flow matching.<n>Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment.
  arXiv  Detail & Related papers  (2024-06-01T06:40:22Z)
• Fast Timing-Conditioned Latent Audio Diffusion [8.774733281142021]
  Stable Audio is capable of rendering stereo signals of up to 95 sec at 44.1kHz in 8 sec on an A100 GPU. It is one of the best in two public text-to-music and -audio benchmarks and, differently from state-of-the-art models, can generate music with structure and stereo sounds.
  arXiv  Detail & Related papers  (2024-02-07T13:23:25Z)
• AudioSR: Versatile Audio Super-resolution at Scale [32.36683443201372]
  We introduce a diffusion-based generative model, AudioSR, that is capable of performing robust audio super-resolution on versatile audio types. Specifically, AudioSR can upsample any input audio signal within the bandwidth range of 2kHz to 16kHz to a high-resolution audio signal at 24kHz bandwidth.
  arXiv  Detail & Related papers  (2023-09-13T21:00:09Z)
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models [65.18102159618631]
  multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. We propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps.
  arXiv  Detail & Related papers  (2023-01-30T04:44:34Z)
• AudioGen: Textually Guided Audio Generation [116.57006301417306]
  We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive model that generates audio samples conditioned on text inputs.
  arXiv  Detail & Related papers  (2022-09-30T10:17:05Z)
• SoundStream: An End-to-End Neural Audio Codec [78.94923131038682]
  We present SoundStream, a novel neural audio system that can efficiently compress speech, music and general audio. SoundStream relies on a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. We are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency.
  arXiv  Detail & Related papers  (2021-07-07T15:45:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.