LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training

• URL: http://arxiv.org/abs/2606.30642v1
• Date: Mon, 29 Jun 2026 17:59:20 GMT
• Title: LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training
• Abstract summary: Full-length song generation must preserve coherence and musicality, render detailed vocal and accompaniment acoustics, and follow lyrics and prompts.<n>We present LeVo 2, a hybrid LLM-Diffusion framework for controllable full-length song generation.
• Score: 42.48917906211788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Full-length song generation must preserve coherence and musicality, render detailed vocal and accompaniment acoustics, and follow lyrics and prompts. Existing language model-based systems face a structural trade-off: mixed-token modeling preserves vocal-instrument coordination but obscures track-specific details, whereas dual-track prediction improves acoustics but requires longer sequences and weakens global planning. We present LeVo 2, a hybrid LLM-Diffusion framework for controllable full-length song generation. LeVo 2 formulates this trade-off as hierarchical modeling: LeLM first predicts mixed tokens for semantic planning, then predicts vocal and accompaniment tokens in parallel for track-specific refinement, while a diffusion-based Music Codec reconstructs full-length waveforms. A central contribution of this extended version is an aesthetics-guided training schedule for alignment. During pre-training, an automated music aesthetic evaluation framework assigns musicality-tier conditions to large-scale data, providing musicality priors before preference alignment. Progressive post-training applies SFT, large-scale offline DPO, and closed-loop semi-online DPO to separately improve generation quality, controllability, and musicality. Modular extension then trains the Track-Specific LM for acoustic refinement while preserving the aligned semantic planner. This schedule separates musicality learning, controllability alignment, and acoustic refinement, mitigating optimization conflict and the limitations of static offline preference pairs. Expert listening tests and objective evaluations show that LeVo 2 outperforms open-source baselines across six subjective dimensions, and approaches leading commercial systems on several listening metrics. Ablations validate the effects of the training strategy, aesthetics guidance, scaling, and hierarchical architecture.

Related papers

• VIBE: Video Instruction-aligned Background music gEneration [58.326977744404566]
  Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations.<n>We introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages Conditioning Connection.
  arXiv  Detail & Related papers  (2026-08-31T01:19:55Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• Khala: Scaling Acoustic Token Language Models Toward High-Fidelity Music Generation [31.997336001394284]
  A common design pattern in high-quality music generation is to handle structure and fidelity in different representation spaces.<n>In this work, we explore an alternative view: both may be progressively modeled within a single deep acoustic-token hierarchy.<n>Our results suggest that high-quality music generation can be effectively pursued without separating structure and fidelity into heterogeneous representation spaces.
  arXiv  Detail & Related papers  (2026-05-03T09:13:20Z)
• YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidance [16.462715982402884]
  Singing Voice Synthesis (SVS) remains constrained in practical deployment due to its strong dependence on accurate phoneme-level alignment.<n>We propose a melody-driven SVS framework capable of synthesizing arbitrary lyrics following any reference melody.<n>Our method builds on a Diffusion Transformer (DiT) architecture, enhanced with a dedicated melody extraction module.
  arXiv  Detail & Related papers  (2025-12-04T13:25:33Z)
• LeVo: High-Quality Song Generation with Multi-Preference Alignment [47.965028296133426]
  We introduce LeVo, a language model based framework consisting of LeLM and Music Codec.<n>LeVo is capable of parallel modeling of two types of tokens: mixed tokens, which represent the combined audio of vocals and accompaniment.<n>It employs two decoder-only transformers and a modular extension training strategy to prevent interference between different token types.
  arXiv  Detail & Related papers  (2025-06-09T07:57:24Z)
• FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing [81.3306413498174]
  Movie Dubbing aims to convert scripts into speeches that align with the given movie clip in both temporal and emotional aspects.<n>Existing methods focus primarily on reducing the word error rate while ignoring the importance of lip-sync and acoustic quality.<n>We propose a large language model (LLM) based flow matching architecture for dubbing, named FlowDubber.
  arXiv  Detail & Related papers  (2025-05-02T13:30:19Z)
• Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization [19.27890803128116]
  We present a unified framework for automatic multitrack music arrangement.<n>At its core is a segment-level reconstruction objective operating on token-level disentangled content and style.<n>To support track-wise modeling, we introduce REMI-z, a structured tokenization scheme for multitrack symbolic music.
  arXiv  Detail & Related papers  (2024-08-27T16:18:51Z)
• DITTO: Diffusion Inference-Time T-Optimization for Music Generation [49.90109850026932]
  Diffusion Inference-Time T-Optimization (DITTO) is a frame-work for controlling pre-trained text-to-music diffusion models at inference-time. We demonstrate a surprisingly wide-range of applications for music generation including inpainting, outpainting, and looping as well as intensity, melody, and musical structure control.
  arXiv  Detail & Related papers  (2024-01-22T18:10:10Z)
• Continuous Melody Generation via Disentangled Short-Term Representations and Structural Conditions [14.786601824794369]
  We present a model for composing melodies given a user specified symbolic scenario combined with a previous music context. Our model is capable of generating long melodies by regarding 8-beat note sequences as basic units, and shares consistent rhythm pattern structure with another specific song. Results show that the music generated by our model tends to have salient repetition structures, rich motives, and stable rhythm patterns.
  arXiv  Detail & Related papers  (2020-02-05T06:23:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.