Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems

• URL: http://arxiv.org/abs/2605.09259v1
• Date: Sun, 10 May 2026 02:08:07 GMT
• Title: Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
• Abstract summary: Timbre transfer aims to modify the timbral identity of a musical recording while preserving the original melody and rhythm.<n>This paper proposes the first system for flexible per-stem timbre transfer directly from a polyphonic mixture.
• Score: 4.026288999965451
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Timbre transfer aims to modify the timbral identity of a musical recording while preserving the original melody and rhythm. While single-instrument timbre transfer has made substantial progress, existing approaches to multi-instrument settings rely on separate-then-transfer pipelines that propagate source separation artifacts and produce incoherent synthesized timbres across stems. This paper proposes MixtureTT, to the best of our knowledge the first system for flexible per-stem timbre transfer directly from a polyphonic mixture. Given a mixture and a separate timbre reference for each target voice, MixtureTT jointly transfers all stems to the specified instruments through a shared diffusion process. Modeling the dependencies across the per-stem content and cross-stem harmonic, the proposed joint stem diffusion transformer eliminates cascaded separation error, reduces inference cost by a factor equal to the number of stems, and yields more coherent multi-stem outputs. Despite operating under a strictly harder input condition, evaluations on the SATB choral dataset show that MixtureTT outperforms single-instrument baselines on both objective and subjective metrics demonstrating the necessity of dedicated multi-instrument timbre transfer over the naive separate-then-transfer pipelines. As a result, this work confirms that the cross-stem modeling is essential for mixture-level timbre transfer as the proposed joint setting consistently exceeds an equivalent single-stem ablation.

Related papers

• Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference [58.189320101488725]
  DLLMs promise fast non-autoregressive inference but suffer a severe quality-speed trade-off in parallel decoding.<n>We address this by integrating continuous representations into the discrete decoding process, as they preserve rich inter-position dependency.<n>We propose ReMix, a framework that introduces a novel Continuous Mixing State as an intermediate between the initial masked state and the final decoded token state.
  arXiv  Detail & Related papers  (2026-02-26T11:08:11Z)
• Stemphonic: All-at-once Flexible Multi-stem Music Generation [15.126857537352182]
  Music stem generation offers greater user control and better alignment with musician.<n>We propose Stemphonic, a diffusion-/flow-based framework that generates a variable set of synchronized stems in one inference pass.<n>We show that Stemphonic produces higher-quality outputs while accelerating the full mix generation process by 25 to 50%.
  arXiv  Detail & Related papers  (2026-02-10T15:30:12Z)
• Diffusion-Guided Mask-Consistent Paired Mixing for Endoscopic Image Segmentation [57.37991748282666]
  We propose a paired, diffusion-guided paradigm that fuses the strengths of sample mixing and diffusion synthesis.<n>For each real image, a synthetic counterpart is generated under the same mask and the pair is used as a controllable input for Mask-Consistent Paired Mixing (MCPMix)<n>This produces a continuous family of intermediate samples that smoothly bridges synthetic and real appearances under shared geometry.
  arXiv  Detail & Related papers  (2025-11-05T06:14:19Z)
• High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling [65.02357548201188]
  We propose DAVIS, a Diffusion-based Audio-VIsual Separation framework that solves the audio-visual sound source separation task through generative learning.<n>Our framework operates by synthesizing the desired separated sound spectrograms directly from a noise distribution, conditioned concurrently on the mixed audio input and associated visual information.
  arXiv  Detail & Related papers  (2025-09-26T08:46:00Z)
• Unleashing the Power of Natural Audio Featuring Multiple Sound Sources [54.38251699625379]
  Universal sound separation aims to extract clean audio tracks corresponding to distinct events from mixed audio.<n>We propose ClearSep, a framework that employs a data engine to decompose complex naturally mixed audio into multiple independent tracks.<n>In experiments, ClearSep achieves state-of-the-art performance across multiple sound separation tasks.
  arXiv  Detail & Related papers  (2025-04-24T17:58:21Z)
• Separate This, and All of these Things Around It: Music Source Separation via Hyperellipsoidal Queries [53.30852012059025]
  Music source separation is an audio-to-audio retrieval task.<n>Recent work in music source separation has begun to challenge the fixed-stem paradigm.<n>We propose the use of hyperellipsoidal regions as queries to allow for an intuitive yet easily parametrizable approach to specifying both the target (location) and its spread.
  arXiv  Detail & Related papers  (2025-01-27T16:13:50Z)
• Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer [23.167490364656654]
  Music timbre transfer involves modifying the timbral characteristics of an audio signal while preserving its melodic structure.<n>We propose a novel method based on dual diffusion bridges, trained using the CocoChorales dataset.<n> Experimental results demonstrate that our method both achieves better Fr'echet Audio Distance (FAD) and melody preservation, as reflected by lower pitch distances (DPD) compared to VAEGAN and GFB.
  arXiv  Detail & Related papers  (2024-09-09T22:16:48Z)
• DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulation [21.06957311285177]
  DisMix is a generative framework in which the pitch and timbre representations act as building blocks for constructing the melody and instrument of a source. By manipulating the representations, our model samples mixtures with novel combinations of pitch and timbre of the constituent instruments. We can jointly learn the disentangled pitch-timbre representations and a latent diffusion transformer that reconstructs the mixture conditioned on the set of source-level representations.
  arXiv  Detail & Related papers  (2024-08-20T12:56:49Z)
• Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects [23.29395422386749]
  We propose an end-to-end music mixing style transfer system that converts the mixing style of an input multitrack to that of a reference song. This is achieved with an encoder pre-trained with a contrastive objective to extract only audio effects related information from a reference music recording.
  arXiv  Detail & Related papers  (2022-11-04T03:45:17Z)
• Deep Multimodal Fusion by Channel Exchanging [87.40768169300898]
  This paper proposes a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network.
  arXiv  Detail & Related papers  (2020-11-10T09:53:20Z)
• Vector-Quantized Timbre Representation [53.828476137089325]
  This paper targets a more flexible synthesis of an individual timbre by learning an approximate decomposition of its spectral properties with a set of generative features. We introduce an auto-encoder with a discrete latent space that is disentangled from loudness in order to learn a quantized representation of a given timbre distribution. We detail results for translating audio between orchestral instruments and singing voice, as well as transfers from vocal imitations to instruments.
  arXiv  Detail & Related papers  (2020-07-13T12:35:45Z)
• Unsupervised Sound Separation Using Mixture Invariant Training [38.0680944898427]
  We show that MixIT can achieve competitive performance compared to supervised methods on speech separation. In particular, we significantly improve reverberant speech separation performance by incorporating reverberant mixtures.
  arXiv  Detail & Related papers  (2020-06-23T02:22:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.