Libretto: Giving LLM Agents a Sense of Musical Structure

• URL: http://arxiv.org/abs/2606.22708v1
• Date: Sun, 21 Jun 2026 22:56:07 GMT
• Title: Libretto: Giving LLM Agents a Sense of Musical Structure
• Abstract summary: We introduce Libretto, a framework for symbolic music generation and revision.<n>Libretto evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation.<n>Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.
• Score: 4.144744763257738
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative music systems can now produce impressive audio from text prompts, but audio outputs are difficult to inspect, edit, and diagnose as musical structure. We introduce Libretto, an agent-facing framework for symbolic music generation and revision. Libretto uses an LLM-native grammar with explicit onset slots, voices, and bar-level organization, then evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation. The same structural axes support retrieval, diagnosis, copy-risk control, and iterative self-revision. Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.

Related papers

• Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering [36.032642952445414]
  We present a unified song generation framework capable of producing high-quality full-length music from lyrics, text descriptions, and musical attributes.<n>The proposed framework supports three tasks: Lyrics-to-Song Generation, which generates complete songs from text descriptions, lyrics, and musical attributes; Instrumental Music Generation, which creates music without vocals; and Cover Song Generation, which reinterprets existing songs with different styles while preserving melodic content.
  arXiv  Detail & Related papers  (2026-07-22T15:11:46Z)
• A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models [0.0]
  We present a real-time musical interface that converts natural-language scene descriptions into evolving procedural soundscapes.<n>A performer types a prompt such as "warm jazz cafe at midnight" and steers it through direct parameter adjustments.<n>We evaluate text-audio semantic alignment using LAION-CLAP on held-out prompts as a technical proxy.
  arXiv  Detail & Related papers  (2026-07-01T01:21:42Z)
• Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding [0.9886053654565675]
  Symbolic music evaluation for large language models remains fragmented across representations, datasets, and metrics.<n>We introduce LilyBench, a LilyPond-based benchmark that jointly evaluates symbolic music generation and music understanding on the same family of open-weight LLMs.
  arXiv  Detail & Related papers  (2026-06-07T16:32:59Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• Communicating Sound Through Natural Language [20.119057003948004]
  We introduce lexical acoustic coding (LAC), a framework in which pre-trained LLM sender and receiver agents transmit sound through natural language.<n>Under fixed system prompts, the agents write their own analysis and synthesis code, communicating only through a lexical sentence.<n>We frame LAC as a finite-rate lossy quantizer, exposing trade-offs between vocabulary size, rate, and fidelity.
  arXiv  Detail & Related papers  (2026-05-09T07:25:54Z)
• VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models [45.70355425430011]
  We present VocalParse, a unified singing voice transcription (SVT) model built upon a Large Audio Language Model (LALM)<n>Our novel contribution is to introduce an interleaved prompting formulation that jointly models lyrics, melody, and word-note correspondence, yielding a generated sequence that directly maps to a structured musical score.<n> Experiments demonstrate that VocalParse achieves state-of-the-art SVT performance on multiple singing datasets.
  arXiv  Detail & Related papers  (2026-05-06T08:03:31Z)
• Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control [66.46754271097555]
  We release a fully open-source system for long-form song generation with fine-grained style conditioning.<n>The dataset consists of 116k fully licensed synthetic songs with automatically generated lyrics and style descriptions.<n>We train Muse via single-stage supervised finetuning of a Qwen-based language model extended with discrete audio tokens.
  arXiv  Detail & Related papers  (2026-01-07T14:40:48Z)
• Discovering "Words" in Music: Unsupervised Learning of Compositional Sparse Code for Symbolic Music [50.87225308217594]
  This paper presents an unsupervised machine learning algorithm that identifies recurring patterns -- referred to as music-words'' -- from symbolic music data.<n>We formulate the task of music-word discovery as a statistical optimization problem and propose a two-stage Expectation-Maximization (EM)-based learning framework.
  arXiv  Detail & Related papers  (2025-09-29T11:10:57Z)
• Large Language Models' Internal Perception of Symbolic Music [3.9901365062418317]
  Large language models (LLMs) excel at modeling relationships between strings in natural language.<n>This paper investigates how LLMs represent musical concepts by generating symbolic music data from textual prompts.
  arXiv  Detail & Related papers  (2025-07-17T05:48:45Z)
• MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation [19.878013881045817]
  MusiConGen is a temporally-conditioned Transformer-based text-to-music model. It integrates automatically-extracted rhythm and chords as the condition signal. We show that MusiConGen can generate realistic backing track music that aligns well with the specified conditions.
  arXiv  Detail & Related papers  (2024-07-21T05:27:53Z)
• SongComposer: A Large Language Model for Lyric and Melody Generation in Song Composition [82.38021790213752]
  SongComposer is a music-specialized large language model (LLM)<n>It integrates the capability of simultaneously composing melodies into LLMs by leveraging three key innovations.<n>It outperforms advanced LLMs in tasks such as lyric-to-melody generation, melody-to-lyric generation, song continuation, and text-to-song creation.<n>We will release SongCompose, a large-scale dataset for training, containing paired lyrics and melodies in Chinese and English.
  arXiv  Detail & Related papers  (2024-02-27T16:15:28Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)
• Vector-Quantized Timbre Representation [53.828476137089325]
  This paper targets a more flexible synthesis of an individual timbre by learning an approximate decomposition of its spectral properties with a set of generative features. We introduce an auto-encoder with a discrete latent space that is disentangled from loudness in order to learn a quantized representation of a given timbre distribution. We detail results for translating audio between orchestral instruments and singing voice, as well as transfers from vocal imitations to instruments.
  arXiv  Detail & Related papers  (2020-07-13T12:35:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.