Long-form music generation with latent diffusion

• URL: http://arxiv.org/abs/2404.10301v2
• Date: Mon, 29 Jul 2024 14:52:26 GMT
• Title: Long-form music generation with latent diffusion
• Authors: Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, Jordi Pons,
• Abstract summary: We show that by training a generative model on long temporal contexts it is possible to produce long-form music of up to 4m45s. We obtains state-of-the-art generations according to metrics on audio quality and prompt alignment, and subjective tests reveal that it produces full-length music with coherent structure.
• Score: 8.799402694043955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio-based generative models for music have seen great strides recently, but so far have not managed to produce full-length music tracks with coherent musical structure from text prompts. We show that by training a generative model on long temporal contexts it is possible to produce long-form music of up to 4m45s. Our model consists of a diffusion-transformer operating on a highly downsampled continuous latent representation (latent rate of 21.5Hz). It obtains state-of-the-art generations according to metrics on audio quality and prompt alignment, and subjective tests reveal that it produces full-length music with coherent structure.

Related papers

• MusicFlow: Cascaded Flow Matching for Text Guided Music Generation [53.63948108922333]
  MusicFlow is a cascaded text-to-music generation model based on flow matching. We leverage masked prediction as the training objective, enabling the model to generalize to other tasks such as music infilling and continuation.
  arXiv  Detail & Related papers  (2024-10-27T15:35:41Z)
• CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation [17.41880273107978]
  Contrastive Long-form Language-Audio Pretraining (textbfCoLLAP) We propose Contrastive Long-form Language-Audio Pretraining (textbfCoLLAP) to significantly extend the perception window for both the input audio (up to 5 minutes) and the language descriptions (exceeding 250 words) We collect 51.3K audio-text pairs derived from the large-scale AudioSet training dataset, where the average audio length reaches 288 seconds.
  arXiv  Detail & Related papers  (2024-10-03T07:46:51Z)
• Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models [0.0]
  "Diff-A-Riff" is a Latent Diffusion Model designed to generate high-quality instrumentals adaptable to any musical context. It produces 48kHz pseudo-stereo audio while significantly reducing inference time and memory usage.
  arXiv  Detail & Related papers  (2024-06-12T16:34:26Z)
• Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization [70.13218512896032]
  Generation of audio from text prompts is an important aspect of such processes in the music and film industry. Our hypothesis is focusing on how these aspects of audio generation could improve audio generation performance in the presence of limited data. We synthetically create a preference dataset where each prompt has a winner audio output and some loser audio outputs for the diffusion model to learn from.
  arXiv  Detail & Related papers  (2024-04-15T17:31:22Z)
• Fast Timing-Conditioned Latent Audio Diffusion [8.774733281142021]
  Stable Audio is capable of rendering stereo signals of up to 95 sec at 44.1kHz in 8 sec on an A100 GPU. It is one of the best in two public text-to-music and -audio benchmarks and, differently from state-of-the-art models, can generate music with structure and stereo sounds.
  arXiv  Detail & Related papers  (2024-02-07T13:23:25Z)
• Controllable Music Production with Diffusion Models and Guidance Gradients [3.187381965457262]
  We demonstrate how conditional generation from diffusion models can be used to tackle a variety of realistic tasks in 44.1kHz stereo audio. The scenarios we consider include continuation, inpainting and regeneration of musical audio, the creation of smooth transitions between two different music tracks, and the transfer of desired stylistic characteristics to existing audio clips.
  arXiv  Detail & Related papers  (2023-11-01T16:01:01Z)
• V2Meow: Meowing to the Visual Beat via Video-to-Music Generation [47.076283429992664]
  V2Meow is a video-to-music generation system capable of producing high-quality music audio for a diverse range of video input types. It synthesizes high-fidelity music audio waveforms solely by conditioning on pre-trained general-purpose visual features extracted from video frames.
  arXiv  Detail & Related papers  (2023-05-11T06:26:41Z)
• Noise2Music: Text-conditioned Music Generation with Diffusion Models [73.74580231353684]
  We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. We find that the generated audio is not only able to faithfully reflect key elements of the text prompt such as genre, tempo, instruments, mood, and era. Pretrained large language models play a key role in this story -- they are used to generate paired text for the audio of the training set and to extract embeddings of the text prompts ingested by the diffusion models.
  arXiv  Detail & Related papers  (2023-02-08T07:27:27Z)
• Mo\^usai: Text-to-Music Generation with Long-Context Latent Diffusion [27.567536688166776]
  We bridge text and music via a text-to-music generation model that is highly efficient, expressive, and can handle long-term structure. Specifically, we develop Mousai, a cascading two-stage latent diffusion model that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions.
  arXiv  Detail & Related papers  (2023-01-27T14:52:53Z)
• AudioLM: a Language Modeling Approach to Audio Generation [59.19364975706805]
  We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure. We demonstrate how our approach extends beyond speech by generating coherent piano music continuations.
  arXiv  Detail & Related papers  (2022-09-07T13:40:08Z)
• Incorporating Music Knowledge in Continual Dataset Augmentation for Music Generation [69.06413031969674]
  Aug-Gen is a method of dataset augmentation for any music generation system trained on a resource-constrained domain. We apply Aug-Gen to Transformer-based chorale generation in the style of J.S. Bach, and show that this allows for longer training and results in better generative output.
  arXiv  Detail & Related papers  (2020-06-23T21:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.