MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models

• URL: http://arxiv.org/abs/2402.06178v3
• Date: Tue, 28 May 2024 16:47:25 GMT
• Title: MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models
• Authors: Yixiao Zhang, Yukara Ikemiya, Gus Xia, Naoki Murata, Marco A. Martínez-Ramírez, Wei-Hsiang Liao, Yuki Mitsufuji, Simon Dixon,
• Abstract summary: This paper introduces a novel approach to the editing of music generated by text-to-music models. Our method transforms text editing to textitlatent space manipulation while adding an extra constraint to enforce consistency. Experimental results demonstrate superior performance over both zero-shot and certain supervised baselines in style and timbre transfer evaluations.
• Score: 24.582948932985726
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent advances in text-to-music generation models have opened new avenues in musical creativity. However, music generation usually involves iterative refinements, and how to edit the generated music remains a significant challenge. This paper introduces a novel approach to the editing of music generated by such models, enabling the modification of specific attributes, such as genre, mood and instrument, while maintaining other aspects unchanged. Our method transforms text editing to \textit{latent space manipulation} while adding an extra constraint to enforce consistency. It seamlessly integrates with existing pretrained text-to-music diffusion models without requiring additional training. Experimental results demonstrate superior performance over both zero-shot and certain supervised baselines in style and timbre transfer evaluations. Additionally, we showcase the practical applicability of our approach in real-world music editing scenarios.

Related papers

• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation [75.86473375730392]
  SongGen is a fully open-source, single-stage auto-regressive transformer for controllable song generation. It supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately. To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline.
  arXiv  Detail & Related papers  (2025-02-18T18:52:21Z)
• ImprovNet: Generating Controllable Musical Improvisations with Iterative Corruption Refinement [6.873190001575463]
  ImprovNet is a transformer-based architecture that generates expressive and controllable musical improvisations. It can perform cross-genre and intra-genre improvisations, harmonize melodies with genre-specific styles, and execute short prompt continuation and infilling tasks.
  arXiv  Detail & Related papers  (2025-02-06T21:45:38Z)
• MusicFlow: Cascaded Flow Matching for Text Guided Music Generation [53.63948108922333]
  MusicFlow is a cascaded text-to-music generation model based on flow matching. We leverage masked prediction as the training objective, enabling the model to generalize to other tasks such as music infilling and continuation.
  arXiv  Detail & Related papers  (2024-10-27T15:35:41Z)
• Efficient Fine-Grained Guidance for Diffusion-Based Symbolic Music Generation [14.156461396686248]
  We introduce an efficient Fine-Grained Guidance (FGG) approach within diffusion models. FGG guides the diffusion models to generate music that aligns more closely with the control and intent of expert composers. This approach empowers diffusion models to excel in advanced applications such as improvisation, and interactive music creation.
  arXiv  Detail & Related papers  (2024-10-11T00:41:46Z)
• MuseBarControl: Enhancing Fine-Grained Control in Symbolic Music Generation through Pre-Training and Counterfactual Loss [51.85076222868963]
  We introduce a pre-training task designed to link control signals directly with corresponding musical tokens. We then implement a novel counterfactual loss that promotes better alignment between the generated music and the control prompts.
  arXiv  Detail & Related papers  (2024-07-05T08:08:22Z)
• MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models [57.47799823804519]
  We are inspired by how musicians compose music not just from a movie script, but also through visualizations. We propose MeLFusion, a model that can effectively use cues from a textual description and the corresponding image to synthesize music. Our exhaustive experimental evaluation suggests that adding visual information to the music synthesis pipeline significantly improves the quality of generated music.
  arXiv  Detail & Related papers  (2024-06-07T06:38:59Z)
• Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning [24.6866990804501]
  Instruct-MusicGen is a novel approach that finetunes a pretrained MusicGen model to efficiently follow editing instructions. Remarkably, Instruct-MusicGen only introduces 8% new parameters to the original MusicGen model and only trains for 5K steps.
  arXiv  Detail & Related papers  (2024-05-28T17:27:20Z)
• FateZero: Fusing Attentions for Zero-shot Text-based Video Editing [104.27329655124299]
  We propose FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask. Our method is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model.
  arXiv  Detail & Related papers  (2023-03-16T17:51:13Z)
• ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models [67.66825818489406]
  This paper introduces a text-to-waveform music generation model, underpinned by the utilization of diffusion models. Our methodology hinges on the innovative incorporation of free-form textual prompts as conditional factors to guide the waveform generation process. We demonstrate that our generated music in the waveform domain outperforms previous works by a large margin in terms of diversity, quality, and text-music relevance.
  arXiv  Detail & Related papers  (2023-02-09T06:27:09Z)
• Actions Speak Louder than Listening: Evaluating Music Style Transfer based on Editing Experience [4.986422167919228]
  We propose an editing test to evaluate users' editing experience of music generation models in a systematic way. Results on two target styles indicate that the improvement over the baseline model can be reflected by the editing test quantitatively.
  arXiv  Detail & Related papers  (2021-10-25T12:20:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.