MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners

• URL: http://arxiv.org/abs/2506.18729v2
• Date: Tue, 24 Jun 2025 15:53:56 GMT
• Title: MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners
• Abstract summary: MuseControlLite is designed to fine-tune text-to-music generation models for precise conditioning.<n> positional embeddings are critical when the condition of interest is a function of time.
• Score: 16.90741849156418
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose MuseControlLite, a lightweight mechanism designed to fine-tune text-to-music generation models for precise conditioning using various time-varying musical attributes and reference audio signals. The key finding is that positional embeddings, which have been seldom used by text-to-music generation models in the conditioner for text conditions, are critical when the condition of interest is a function of time. Using melody control as an example, our experiments show that simply adding rotary positional embeddings to the decoupled cross-attention layers increases control accuracy from 56.6% to 61.1%, while requiring 6.75 times fewer trainable parameters than state-of-the-art fine-tuning mechanisms, using the same pre-trained diffusion Transformer model of Stable Audio Open. We evaluate various forms of musical attribute control, audio inpainting, and audio outpainting, demonstrating improved controllability over MusicGen-Large and Stable Audio Open ControlNet at a significantly lower fine-tuning cost, with only 85M trainble parameters. Source code, model checkpoints, and demo examples are available at: https://musecontrollite.github.io/web/.

Related papers

• Sound Sparks Motion: Audio and Text Tuning for Video Editing [53.136757756110626]
  We introduce Sound Sparks Motion, a training-free framework that enables motion editing in an audio-visual video generation model.<n>Sound Sparks Motion tunes its internal multimodal conditioning signals at test time.<n>Our results highlight multimodal conditioning tuning as a promising direction for motion-aware video editing.
  arXiv  Detail & Related papers  (2026-05-14T18:20:50Z)
• Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control [66.46754271097555]
  We release a fully open-source system for long-form song generation with fine-grained style conditioning.<n>The dataset consists of 116k fully licensed synthetic songs with automatically generated lyrics and style descriptions.<n>We train Muse via single-stage supervised finetuning of a Qwen-based language model extended with discrete audio tokens.
  arXiv  Detail & Related papers  (2026-01-07T14:40:48Z)
• InstructAudio: Unified speech and music generation with natural language instruction [52.76518112649456]
  InstructAudio is a unified framework that enables instruction-based control of acoustic attributes.<n>It supports expressive speech, music, and dialogue generation in English and Chinese.
  arXiv  Detail & Related papers  (2025-11-23T15:15:21Z)
• Steering Autoregressive Music Generation with Recursive Feature Machines [43.475981527010276]
  MusicRFM is a framework that adapts Recursive Feature Machines (RFMs) to enable fine-grained, interpretable control over frozen, pre-trained music models.<n>RFMs analyze a model's internal gradients to produce interpretable "concept directions"<n>We present advanced mechanisms for this control, including dynamic, time-varying schedules and methods for the simultaneous enforcement of multiple musical properties.
  arXiv  Detail & Related papers  (2025-10-21T23:23:14Z)
• LiLAC: A Lightweight Latent ControlNet for Musical Audio Generation [0.0]
  ControlNet enables attaching external controls to a pre-trained generative model by cloning and fine-tuning its encoder on new conditionings.<n>We propose a lightweight, modular architecture that considerably reduces parameter count while matching ControlNet in audio quality and condition adherence.
  arXiv  Detail & Related papers  (2025-06-13T05:39:50Z)
• Versatile Framework for Song Generation with Prompt-based Control [50.359999116420084]
  VersBand is a framework for synthesizing high-quality, aligned songs with prompt-based control.<n>VocalBand, a decoupled model, leverages the flow-matching method for generating singing styles, pitches, and mel-spectrograms.<n>AccompBand, a flow-based transformer model, incorporates the Band-MOE, selecting suitable experts for enhanced quality, alignment, and control.<n>Two generation models, LyricBand for lyrics and MelodyBand for melodies, contribute to the comprehensive multi-task song generation system.
  arXiv  Detail & Related papers  (2025-04-27T01:00:06Z)
• MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation [19.878013881045817]
  MusiConGen is a temporally-conditioned Transformer-based text-to-music model. It integrates automatically-extracted rhythm and chords as the condition signal. We show that MusiConGen can generate realistic backing track music that aligns well with the specified conditions.
  arXiv  Detail & Related papers  (2024-07-21T05:27:53Z)
• BandControlNet: Parallel Transformers-based Steerable Popular Music Generation with Fine-Grained Spatiotemporal Features [19.284531698181116]
  BandControlNet is designed to tackle the multiple music sequences and generate high-quality music samples conditioned to the giventemporal control features. The proposed BandControlNet outperforms other conditional music generation models on most objective metrics in terms of fidelity and inference speed. The subjective evaluations show trained on short datasets can generate music with comparable quality to state-of-the-art models, while outperforming significantly using BandControlNet.
  arXiv  Detail & Related papers  (2024-07-15T06:33:25Z)
• MuseBarControl: Enhancing Fine-Grained Control in Symbolic Music Generation through Pre-Training and Counterfactual Loss [51.85076222868963]
  We introduce a pre-training task designed to link control signals directly with corresponding musical tokens. We then implement a novel counterfactual loss that promotes better alignment between the generated music and the control prompts.
  arXiv  Detail & Related papers  (2024-07-05T08:08:22Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language.<n>We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation.<n>Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• Content-based Controls For Music Large Language Modeling [6.17674772485321]
  Coco-Mulla is a content-based control method for music large language modeling. It uses a parameter-efficient fine-tuning (PEFT) method tailored for Transformer-based audio models. Our approach achieves high-quality music generation with low-resource semi-supervised learning.
  arXiv  Detail & Related papers  (2023-10-26T05:24:38Z)
• From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion [84.138804145918]
  Deep generative models can generate high-fidelity audio conditioned on various types of representations. These models are prone to generate audible artifacts when the conditioning is flawed or imperfect. We propose a high-fidelity multi-band diffusion-based framework that generates any type of audio modality from low-bitrate discrete representations.
  arXiv  Detail & Related papers  (2023-08-02T22:14:29Z)
• Anticipatory Music Transformer [60.15347393822849]
  We introduce anticipation: a method for constructing a controllable generative model of a temporal point process. We focus on infilling control tasks, whereby the controls are a subset of the events themselves. We train anticipatory infilling models using the large and diverse Lakh MIDI music dataset.
  arXiv  Detail & Related papers  (2023-06-14T16:27:53Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.