MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction

• URL: http://arxiv.org/abs/2505.23305v1
• Date: Thu, 29 May 2025 10:04:24 GMT
• Title: MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
• Abstract summary: MGE-LDM is a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation.<n>By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources.
• Score: 15.838868785353908
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation. Unlike prior approaches constrained to fixed instrument classes, MGE-LDM learns a joint distribution over full mixtures, submixtures, and individual stems within a single compact latent diffusion model. At inference, MGE-LDM enables (1) complete mixture generation, (2) partial generation (i.e., source imputation), and (3) text-conditioned extraction of arbitrary sources. By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources. Notably, MGE-LDM can be trained jointly across heterogeneous multi-track datasets (e.g., Slakh2100, MUSDB18, MoisesDB) without relying on predefined instrument categories. Audio samples are available at our project page: https://yoongi43.github.io/MGELDM_Samples/.

Related papers

• MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation [26.862922663735347]
  MUNI is an end-to-end multimodal latent diffusion framework for any-to-any generation.<n>It unifies subset-conditioned cross-modal generation and unconditional joint sampling through a shared latent.<n>MUNI rests on two complementary contributions, one architectural and one in the training objective.
  arXiv  Detail & Related papers  (2026-06-15T08:46:48Z)
• LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model [77.66516875262963]
  We present textbfLLaDA-o, an effective and length-adaptive omni diffusion model for multimodal understanding and generation.<n>Building on MoD, we introduce a data-centric length adaptation strategy that enables flexible-length decoding in multimodal settings.<n>Experiments show that LLaDA-o achieves state-of-the-art performance among omni-diffusion models on multimodal understanding and generation benchmarks.
  arXiv  Detail & Related papers  (2026-03-01T12:05:06Z)
• Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model [12.393086516044866]
  In this work, we study the potential of diffusion models to advance toward bridging this gap.<n>We focus on generative singing voice separation relying on corresponding pairs of isolated vocals and mixtures for training.<n>To align with creative mixtures, we leverage latent diffusion: the system generates samples encoded in a compact latent space, and subsequently decodes these into audio.
  arXiv  Detail & Related papers  (2025-11-25T16:34:07Z)
• MExD: An Expert-Infused Diffusion Model for Whole-Slide Image Classification [46.89908887119571]
  Whole Slide Image (WSI) classification poses unique challenges due to the vast image size and numerous non-informative regions.<n>We propose MExD, an Expert-Infused Diffusion Model that combines the strengths of a Mixture-of-Experts (MoE) mechanism with a diffusion model for enhanced classification.
  arXiv  Detail & Related papers  (2025-03-16T08:04:17Z)
• Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models [26.373204974010086]
  Multi-Source Diffusion Models (MSDM) allow for compositional musical generation tasks. This paper generalizes MSDM to arbitrary time-domain diffusion models conditioned on text embeddings. We propose an inference procedure enabling the coherent generation of sources and accompaniments.
  arXiv  Detail & Related papers  (2024-03-18T12:08:01Z)
• Learning Joint Latent Space EBM Prior Model for Multi-layer Generator [44.4434704520236]
  We study the fundamental problem of learning multi-layer generator models. We propose an energy-based model (EBM) on the joint latent space over all layers of latent variables. Our experiments demonstrate that the learned model can be expressive in generating high-quality images.
  arXiv  Detail & Related papers  (2023-06-10T00:27:37Z)
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion [144.9653045465908]
  We propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM) Our approach yields promising fusion results in infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2023-03-13T04:06:42Z)
• One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale [36.590918776922905]
  This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Inspired by the unified view, UniDiffuser learns all distributions simultaneously with a minimal modification to the original diffusion model.
  arXiv  Detail & Related papers  (2023-03-12T03:38:39Z)
• Multi-Source Diffusion Models for Simultaneous Music Generation and Separation [17.124189082882395]
  We train our model on Slakh2100, a standard dataset for musical source separation. Our method is the first example of a single model that can handle both generation and separation tasks.
  arXiv  Detail & Related papers  (2023-02-04T23:18:36Z)
• Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models [54.1843419649895]
  We propose a solution based on denoising diffusion probabilistic models (DDPMs) Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task.
  arXiv  Detail & Related papers  (2022-12-01T18:59:55Z)
• Diffusion-based Generative Speech Source Separation [27.928990101986862]
  We propose DiffSep, a new single channel source separation method based on score-matching of a differential equation (SDE) Experiments on the WSJ0 2mix dataset demonstrate the potential of the method. The method is also suitable for speech enhancement and shows performance competitive with prior work on the VoiceBank-DEMAND dataset.
  arXiv  Detail & Related papers  (2022-10-31T13:46:55Z)
• Learning with MISELBO: The Mixture Cookbook [62.75516608080322]
  We present the first ever mixture of variational approximations for a normalizing flow-based hierarchical variational autoencoder (VAE) with VampPrior and a PixelCNN decoder network. We explain this cooperative behavior by drawing a novel connection between VI and adaptive importance sampling. We obtain state-of-the-art results among VAE architectures in terms of negative log-likelihood on the MNIST and FashionMNIST datasets.
  arXiv  Detail & Related papers  (2022-09-30T15:01:35Z)
• Meta-Causal Feature Learning for Out-of-Distribution Generalization [71.38239243414091]
  This paper presents a balanced meta-causal learner (BMCL), which includes a balanced task generation module (BTG) and a meta-causal feature learning module (MCFL) BMCL effectively identifies the class-invariant visual regions for classification and may serve as a general framework to improve the performance of the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-08-22T09:07:02Z)
• A new perspective on probabilistic image modeling [92.89846887298852]
  We present a new probabilistic approach for image modeling capable of density estimation, sampling and tractable inference. DCGMMs can be trained end-to-end by SGD from random initial conditions, much like CNNs. We show that DCGMMs compare favorably to several recent PC and SPN models in terms of inference, classification and sampling.
  arXiv  Detail & Related papers  (2022-03-21T14:53:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.