Multi-Source Diffusion Models for Simultaneous Music Generation and Separation

• URL: http://arxiv.org/abs/2302.02257v4
• Date: Mon, 18 Mar 2024 11:39:29 GMT
• Title: Multi-Source Diffusion Models for Simultaneous Music Generation and Separation
• Authors: Giorgio Mariani, Irene Tallini, Emilian Postolache, Michele Mancusi, Luca Cosmo, Emanuele Rodolà,
• Abstract summary: We train our model on Slakh2100, a standard dataset for musical source separation. Our method is the first example of a single model that can handle both generation and separation tasks.
• Score: 17.124189082882395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we define a diffusion-based generative model capable of both music synthesis and source separation by learning the score of the joint probability density of sources sharing a context. Alongside the classic total inference tasks (i.e., generating a mixture, separating the sources), we also introduce and experiment on the partial generation task of source imputation, where we generate a subset of the sources given the others (e.g., play a piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task based on Dirac likelihood functions. We train our model on Slakh2100, a standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the source separation setting. Our method is the first example of a single model that can handle both generation and separation tasks, thus representing a step toward general audio models.

Related papers

• InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion [53.90516061351706]
  We present InterHandGen, a novel framework that learns the generative prior of two-hand interaction. For sampling, we combine anti-penetration and synthesis-free guidance to enable plausible generation. Our method significantly outperforms baseline generative models in terms of plausibility and diversity.
  arXiv  Detail & Related papers  (2024-03-26T06:35:55Z)
• Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models [26.373204974010086]
  Multi-Source Diffusion Models (MSDM) allow for compositional musical generation tasks. This paper generalizes MSDM to arbitrary time-domain diffusion models conditioned on text embeddings. We propose an inference procedure enabling the coherent generation of sources and accompaniments.
  arXiv  Detail & Related papers  (2024-03-18T12:08:01Z)
• Score-based Source Separation with Applications to Digital Communication Signals [72.6570125649502]
  We propose a new method for separating superimposed sources using diffusion-based generative models. Motivated by applications in radio-frequency (RF) systems, we are interested in sources with underlying discrete nature. Our method can be viewed as a multi-source extension to the recently proposed score distillation sampling scheme.
  arXiv  Detail & Related papers  (2023-06-26T04:12:40Z)
• Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation [99.19786288094596]
  We show how the upper bound can be generalized to the case of random generative models. We show state-of-the-art results on 2, 3, 5, 10, and 20 speakers on multiple benchmarks.
  arXiv  Detail & Related papers  (2023-01-25T18:21:51Z)
• Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models [54.1843419649895]
  We propose a solution based on denoising diffusion probabilistic models (DDPMs) Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task.
  arXiv  Detail & Related papers  (2022-12-01T18:59:55Z)
• Diffusion-based Generative Speech Source Separation [27.928990101986862]
  We propose DiffSep, a new single channel source separation method based on score-matching of a differential equation (SDE) Experiments on the WSJ0 2mix dataset demonstrate the potential of the method. The method is also suitable for speech enhancement and shows performance competitive with prior work on the VoiceBank-DEMAND dataset.
  arXiv  Detail & Related papers  (2022-10-31T13:46:55Z)
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning [85.55727213502402]
  We focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. We propose Sample-specific Ensemble of Source Models (SESoM) SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs.
  arXiv  Detail & Related papers  (2022-10-23T01:33:16Z)
• Unsupervised Audio Source Separation Using Differentiable Parametric Source Models [8.80867379881193]
  We propose an unsupervised model-based deep learning approach to musical source separation. A neural network is trained to reconstruct the observed mixture as a sum of the sources. The experimental evaluation on a vocal ensemble separation task shows that the proposed method outperforms learning-free methods.
  arXiv  Detail & Related papers  (2022-01-24T11:05:30Z)
• "Don't quote me on that": Finding Mixtures of Sources in News Articles [85.92467549469147]
  We construct an ontological labeling system for sources based on each source's textitaffiliation and textitrole We build a probabilistic model to infer these attributes for named sources and to describe news articles as mixtures of these sources.
  arXiv  Detail & Related papers  (2021-04-19T21:57:11Z)
• Source Separation with Deep Generative Priors [17.665938343060112]
  We use generative models as priors over the components of a mixture of sources, and noise-annealed Langevin dynamics to sample from the posterior distribution of sources given a mixture. This decouples the source separation problem from generative modeling, enabling us to directly use cutting-edge generative models as priors. The method achieves state-of-the-art performance for MNIST digit separation.
  arXiv  Detail & Related papers  (2020-02-19T00:48:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.