SketchSong: Hierarchical Song Generation with Sketch Planning and Fine-Grained Multi-Track Modeling

• URL: http://arxiv.org/abs/2606.03169v1
• Date: Tue, 02 Jun 2026 05:27:56 GMT
• Title: SketchSong: Hierarchical Song Generation with Sketch Planning and Fine-Grained Multi-Track Modeling
• Abstract summary: SketchSong is a hierarchical song generation framework that addresses issues through song-level sketch planning and fine-grained multi-track modeling.<n>Along the track dimension, SketchSong explicitly models four tracks, i.e., vocals, bass, drums and other instruments.<n>Experiments on song generation benchmarks show that SketchSong consistently outperforms our baseline on both objective metrics and human listening tests.
• Score: 21.874594911334285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent song generation systems can synthesize realistic audio, yet generating complete songs remains challenging for two reasons. First, explicit song-level arrangement planning remains limited in existing methods, so models often need to organize overall arrangement development while generating low-level audio details. This often leads to incoherence in arrangements, such as weak section transitions and limited dynamic progression. Second, coarse modeling of different musical parts obscures their distinct roles and interactions, limiting arrangement richness of generated songs. In this paper, we present SketchSong, a hierarchical song generation framework that addresses these issues through song-level sketch planning and fine-grained multi-track modeling. Along the temporal dimension, SketchSong first predicts a compact sequence of high-level sketch tokens derived from compressed audio representations, and then generates audio tokens conditioned on these sketches. This coarse-to-fine process gives the model an explicit arrangement plan before detailed audio generation. Along the track dimension, SketchSong explicitly models four tracks, i.e., vocals, bass, drums and other instruments. This enables the model to capture the roles and interactions of different musical parts more precisely. Experiments on song generation benchmarks show that SketchSong consistently outperforms our baseline on both objective metrics and human listening tests. Despite not employing additional post-training for preference optimization such as lyrics and text-prompt alignments, SketchSong achieves competitive results against strong, post-trained open-source systems, demonstrating the effectiveness of our overall design.

Related papers

• Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering [36.032642952445414]
  We present a unified song generation framework capable of producing high-quality full-length music from lyrics, text descriptions, and musical attributes.<n>The proposed framework supports three tasks: Lyrics-to-Song Generation, which generates complete songs from text descriptions, lyrics, and musical attributes; Instrumental Music Generation, which creates music without vocals; and Cover Song Generation, which reinterprets existing songs with different styles while preserving melodic content.
  arXiv  Detail & Related papers  (2026-07-22T15:11:46Z)
• LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training [42.48917906211788]
  Full-length song generation must preserve coherence and musicality, render detailed vocal and accompaniment acoustics, and follow lyrics and prompts.<n>We present LeVo 2, a hybrid LLM-Diffusion framework for controllable full-length song generation.
  arXiv  Detail & Related papers  (2026-06-29T17:59:20Z)
• Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control [66.46754271097555]
  We release a fully open-source system for long-form song generation with fine-grained style conditioning.<n>The dataset consists of 116k fully licensed synthetic songs with automatically generated lyrics and style descriptions.<n>We train Muse via single-stage supervised finetuning of a Qwen-based language model extended with discrete audio tokens.
  arXiv  Detail & Related papers  (2026-01-07T14:40:48Z)
• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation [75.86473375730392]
  SongGen is a fully open-source, single-stage auto-regressive transformer for controllable song generation.<n>It supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately.<n>To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline.
  arXiv  Detail & Related papers  (2025-02-18T18:52:21Z)
• Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization [19.27890803128116]
  We present a unified framework for automatic multitrack music arrangement.<n>At its core is a segment-level reconstruction objective operating on token-level disentangled content and style.<n>To support track-wise modeling, we introduce REMI-z, a structured tokenization scheme for multitrack symbolic music.
  arXiv  Detail & Related papers  (2024-08-27T16:18:51Z)
• MuPT: A Generative Symbolic Music Pretrained Transformer [56.09299510129221]
  We explore the application of Large Language Models (LLMs) to the pre-training of music. To address the challenges associated with misaligned measures from different tracks during generation, we propose a Synchronized Multi-Track ABC Notation (SMT-ABC Notation) Our contributions include a series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our training set.
  arXiv  Detail & Related papers  (2024-04-09T15:35:52Z)
• JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation [18.979064278674276]
  JEN-1 Composer is designed to efficiently model marginal, conditional, and joint distributions over multi-track music.<n>We introduce a progressive curriculum training strategy, which gradually escalates the difficulty of training tasks.<n>Our approach demonstrates state-of-the-art performance in controllable and high-fidelity multi-track music synthesis.
  arXiv  Detail & Related papers  (2023-10-29T22:51:49Z)
• Unsupervised Melody-to-Lyric Generation [91.29447272400826]
  We propose a method for generating high-quality lyrics without training on any aligned melody-lyric data. We leverage the segmentation and rhythm alignment between melody and lyrics to compile the given melody into decoding constraints. Our model can generate high-quality lyrics that are more on-topic, singable, intelligible, and coherent than strong baselines.
  arXiv  Detail & Related papers  (2023-05-30T17:20:25Z)
• GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework [58.64512825534638]
  Symbolic music generation aims to create musical notes, which can help users compose music. We introduce a framework known as GETMusic, with GET'' standing for GEnerate music Tracks'' GETScore represents musical notes as tokens and organizes tokens in a 2D structure, with tracks stacked vertically and progressing horizontally over time. Our proposed representation, coupled with the non-autoregressive generative model, empowers GETMusic to generate music with any arbitrary source-target track combinations.
  arXiv  Detail & Related papers  (2023-05-18T09:53:23Z)
• PopMAG: Pop Music Accompaniment Generation [190.09996798215738]
  We propose a novel MUlti-track MIDI representation (MuMIDI) which enables simultaneous multi-track generation in a single sequence. MuMIDI enlarges the sequence length and brings the new challenge of long-term music modeling. We call our system for pop music accompaniment generation as PopMAG.
  arXiv  Detail & Related papers  (2020-08-18T02:28:36Z)
• Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm [42.694266687511906]
  Music SketchNet is a neural network framework that allows users to specify partial musical ideas guiding automatic music generation. We focus on generating the missing measures in incomplete monophonic musical pieces, conditioned on surrounding context. We demonstrate that our model can successfully incorporate user-specified snippets during the generation process.
  arXiv  Detail & Related papers  (2020-08-04T02:49:57Z)
• Continuous Melody Generation via Disentangled Short-Term Representations and Structural Conditions [14.786601824794369]
  We present a model for composing melodies given a user specified symbolic scenario combined with a previous music context. Our model is capable of generating long melodies by regarding 8-beat note sequences as basic units, and shares consistent rhythm pattern structure with another specific song. Results show that the music generated by our model tends to have salient repetition structures, rich motives, and stable rhythm patterns.
  arXiv  Detail & Related papers  (2020-02-05T06:23:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.