MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction

• URL: http://arxiv.org/abs/2505.23305v1
• Date: Thu, 29 May 2025 10:04:24 GMT
• Title: MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
• Authors: Yunkee Chae, Kyogu Lee,
• Abstract summary: MGE-LDM is a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation.<n>By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources.
• Score: 15.838868785353908
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation. Unlike prior approaches constrained to fixed instrument classes, MGE-LDM learns a joint distribution over full mixtures, submixtures, and individual stems within a single compact latent diffusion model. At inference, MGE-LDM enables (1) complete mixture generation, (2) partial generation (i.e., source imputation), and (3) text-conditioned extraction of arbitrary sources. By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources. Notably, MGE-LDM can be trained jointly across heterogeneous multi-track datasets (e.g., Slakh2100, MUSDB18, MoisesDB) without relying on predefined instrument categories. Audio samples are available at our project page: https://yoongi43.github.io/MGELDM_Samples/.

Related papers

• MExD: An Expert-Infused Diffusion Model for Whole-Slide Image Classification [46.89908887119571]
  Whole Slide Image (WSI) classification poses unique challenges due to the vast image size and numerous non-informative regions.<n>We propose MExD, an Expert-Infused Diffusion Model that combines the strengths of a Mixture-of-Experts (MoE) mechanism with a diffusion model for enhanced classification.
  arXiv  Detail & Related papers  (2025-03-16T08:04:17Z)
• Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models [26.373204974010086]
  Multi-Source Diffusion Models (MSDM) allow for compositional musical generation tasks. This paper generalizes MSDM to arbitrary time-domain diffusion models conditioned on text embeddings. We propose an inference procedure enabling the coherent generation of sources and accompaniments.
  arXiv  Detail & Related papers  (2024-03-18T12:08:01Z)
• Learning Joint Latent Space EBM Prior Model for Multi-layer Generator [44.4434704520236]
  We study the fundamental problem of learning multi-layer generator models. We propose an energy-based model (EBM) on the joint latent space over all layers of latent variables. Our experiments demonstrate that the learned model can be expressive in generating high-quality images.
  arXiv  Detail & Related papers  (2023-06-10T00:27:37Z)
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion [144.9653045465908]
  We propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM) Our approach yields promising fusion results in infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2023-03-13T04:06:42Z)
• One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale [36.590918776922905]
  This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Inspired by the unified view, UniDiffuser learns all distributions simultaneously with a minimal modification to the original diffusion model.
  arXiv  Detail & Related papers  (2023-03-12T03:38:39Z)
• Multi-Source Diffusion Models for Simultaneous Music Generation and Separation [17.124189082882395]
  We train our model on Slakh2100, a standard dataset for musical source separation. Our method is the first example of a single model that can handle both generation and separation tasks.
  arXiv  Detail & Related papers  (2023-02-04T23:18:36Z)
• Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models [54.1843419649895]
  We propose a solution based on denoising diffusion probabilistic models (DDPMs) Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task.
  arXiv  Detail & Related papers  (2022-12-01T18:59:55Z)
• Diffusion-based Generative Speech Source Separation [27.928990101986862]
  We propose DiffSep, a new single channel source separation method based on score-matching of a differential equation (SDE) Experiments on the WSJ0 2mix dataset demonstrate the potential of the method. The method is also suitable for speech enhancement and shows performance competitive with prior work on the VoiceBank-DEMAND dataset.
  arXiv  Detail & Related papers  (2022-10-31T13:46:55Z)
• Learning with MISELBO: The Mixture Cookbook [62.75516608080322]
  We present the first ever mixture of variational approximations for a normalizing flow-based hierarchical variational autoencoder (VAE) with VampPrior and a PixelCNN decoder network. We explain this cooperative behavior by drawing a novel connection between VI and adaptive importance sampling. We obtain state-of-the-art results among VAE architectures in terms of negative log-likelihood on the MNIST and FashionMNIST datasets.
  arXiv  Detail & Related papers  (2022-09-30T15:01:35Z)
• Meta-Causal Feature Learning for Out-of-Distribution Generalization [71.38239243414091]
  This paper presents a balanced meta-causal learner (BMCL), which includes a balanced task generation module (BTG) and a meta-causal feature learning module (MCFL) BMCL effectively identifies the class-invariant visual regions for classification and may serve as a general framework to improve the performance of the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-08-22T09:07:02Z)
• A new perspective on probabilistic image modeling [92.89846887298852]
  We present a new probabilistic approach for image modeling capable of density estimation, sampling and tractable inference. DCGMMs can be trained end-to-end by SGD from random initial conditions, much like CNNs. We show that DCGMMs compare favorably to several recent PC and SPN models in terms of inference, classification and sampling.
  arXiv  Detail & Related papers  (2022-03-21T14:53:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.