Content-based Controls For Music Large Language Modeling

• URL: http://arxiv.org/abs/2310.17162v3
• Date: Sun, 06 Oct 2024 21:36:20 GMT
• Title: Content-based Controls For Music Large Language Modeling
• Authors: Liwei Lin, Gus Xia, Junyan Jiang, Yixiao Zhang,
• Abstract summary: Coco-Mulla is a content-based control method for music large language modeling. It uses a parameter-efficient fine-tuning (PEFT) method tailored for Transformer-based audio models. Our approach achieves high-quality music generation with low-resource semi-supervised learning.
• Score: 6.17674772485321
• License:
• Abstract: Recent years have witnessed a rapid growth of large-scale language models in the domain of music audio. Such models enable end-to-end generation of higher-quality music, and some allow conditioned generation using text descriptions. However, the control power of text controls on music is intrinsically limited, as they can only describe music indirectly through meta-data (such as singers and instruments) or high-level representations (such as genre and emotion). We aim to further equip the models with direct and content-based controls on innate music languages such as pitch, chords and drum track. To this end, we contribute Coco-Mulla, a content-based control method for music large language modeling. It uses a parameter-efficient fine-tuning (PEFT) method tailored for Transformer-based audio models. Experiments show that our approach achieved high-quality music generation with low-resource semi-supervised learning, tuning with less than 4% parameters compared to the original model and training on a small dataset with fewer than 300 songs. Moreover, our approach enables effective content-based controls, and we illustrate the control power via chords and rhythms, two of the most salient features of music audio. Furthermore, we show that by combining content-based controls and text descriptions, our system achieves flexible music variation generation and arrangement. Our source codes and demos are available online.

Related papers

• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation [75.86473375730392]
  SongGen is a fully open-source, single-stage auto-regressive transformer for controllable song generation. It supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately. To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline.
  arXiv  Detail & Related papers  (2025-02-18T18:52:21Z)
• MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation [19.878013881045817]
  MusiConGen is a temporally-conditioned Transformer-based text-to-music model. It integrates automatically-extracted rhythm and chords as the condition signal. We show that MusiConGen can generate realistic backing track music that aligns well with the specified conditions.
  arXiv  Detail & Related papers  (2024-07-21T05:27:53Z)
• Read, Watch and Scream! Sound Generation from Text and Video [23.990569918960315]
  Video serves as a conditional control for a text-to-audio generation model. We employ a well-performing text-to-audio model to consolidate the video control. Our method shows superiority in terms of quality, controllability, and training efficiency.
  arXiv  Detail & Related papers  (2024-07-08T01:59:17Z)
• MuseBarControl: Enhancing Fine-Grained Control in Symbolic Music Generation through Pre-Training and Counterfactual Loss [51.85076222868963]
  We introduce a pre-training task designed to link control signals directly with corresponding musical tokens. We then implement a novel counterfactual loss that promotes better alignment between the generated music and the control prompts.
  arXiv  Detail & Related papers  (2024-07-05T08:08:22Z)
• VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling [71.01050359126141]
  We propose VidMuse, a framework for generating music aligned with video inputs. VidMuse produces high-fidelity music that is both acoustically and semantically aligned with the video.
  arXiv  Detail & Related papers  (2024-06-06T17:58:11Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language. We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation. Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls [6.176747724853209]
  Large Language Models (LLMs) have shown promise in generating high-quality music, but their focus on autoregressive generation limits their utility in music editing tasks. We propose a novel approach leveraging a parameter-efficient heterogeneous adapter combined with a masking training scheme. Our method integrates frame-level content-based controls, facilitating track-conditioned music refinement and score-conditioned music arrangement.
  arXiv  Detail & Related papers  (2024-02-14T19:00:01Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)
• Mo\^usai: Text-to-Music Generation with Long-Context Latent Diffusion [27.567536688166776]
  We bridge text and music via a text-to-music generation model that is highly efficient, expressive, and can handle long-term structure. Specifically, we develop Mousai, a cascading two-stage latent diffusion model that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions.
  arXiv  Detail & Related papers  (2023-01-27T14:52:53Z)
• Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task [86.72661027591394]
  We generate complete and semantically consistent symbolic music scores from text descriptions. We explore the efficacy of using publicly available checkpoints for natural language processing in the task of text-to-music generation. Our experimental results show that the improvement from using pre-trained checkpoints is statistically significant in terms of BLEU score and edit distance similarity.
  arXiv  Detail & Related papers  (2022-11-21T07:19:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.