Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding

• URL: http://arxiv.org/abs/2606.08722v1
• Date: Sun, 07 Jun 2026 16:32:59 GMT
• Title: Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding
• Abstract summary: Symbolic music evaluation for large language models remains fragmented across representations, datasets, and metrics.<n>We introduce LilyBench, a LilyPond-based benchmark that jointly evaluates symbolic music generation and music understanding on the same family of open-weight LLMs.
• Score: 0.9886053654565675
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Symbolic music evaluation for large language models remains fragmented across representations, datasets, and metrics. We introduce LilyBench, a LilyPond-based benchmark that jointly evaluates symbolic music generation and music understanding on the same family of open-weight LLMs. The benchmark includes a 200-prompt generation suite and ten understanding tasks adapted from ABC-Eval, covering syntax, metadata prediction, structural sequencing, and music recognition. Generation quality is evaluated using compile rate, MusPy descriptor distributions via Jensen-Shannon similarity, and LilyBERT-based Fréchet Music Distance (FMD). Experiments on four open-weight models show that executable LilyPond generation is achievable in zero-shot settings, while structural understanding tasks remain challenging despite strong performance on composer and genre recognition. Our experiments also reveal systematic disagreements between descriptor-based and embedding-based metrics, suggesting that symbolic music evaluation benefits from metric triangulation rather than single-score ranking. We release the benchmark, prompt bank, and evaluation code to support future research in symbolic music generation and understanding at https://github.com/CSCPadova/lilybench

Related papers

• Libretto: Giving LLM Agents a Sense of Musical Structure [4.144744763257738]
  We introduce Libretto, a framework for symbolic music generation and revision.<n>Libretto evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation.<n>Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.
  arXiv  Detail & Related papers  (2026-06-21T22:56:07Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• BMdataset: A Musicologically Curated LilyPond Dataset [1.0705399532413615]
  BMdataset is a musicologically curated dataset of 393 LilyPond scores (2,646 movements)<n>We introduce LilyBERT, a CodeBERT-based encoder adapted to symbolic music through vocabulary extension.<n>We show that fine-tuning on BMdataset alone outperforms continuous pre-training on the full PDMX corpus for both composer and style classification.
  arXiv  Detail & Related papers  (2026-04-12T13:09:20Z)
• Beyond Musical Descriptors: Extracting Preference-Bearing Intent in Music Queries [9.932656605847727]
  We introduce MusicRecoIntent, a manually annotated corpus of 2,291 Reddit music requests.<n>We label musical descriptors across seven categories with positive, negative, or referential preference-bearing roles.
  arXiv  Detail & Related papers  (2026-02-11T10:52:38Z)
• Factual and Musical Evaluation Metrics for Music Language Models [5.124350492915739]
  Music language models (Music LMs) leverage multimodal representations to answer natural language queries about musical audio recordings.<n>Although Music LMs are reportedly improving, we find that current evaluations fail to capture whether their answers are correct.<n>We propose (1) a better general-purpose evaluation metric for Music LMs adapted to the music domain and (2) a factual evaluation framework to quantify the correctness of a Music LM's responses.
  arXiv  Detail & Related papers  (2025-11-02T18:08:26Z)
• MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding [46.89003337712407]
  We propose MuseTok, a tokenization method for symbolic music.<n>MuseTok employs the residual vector quantized-variational autoencoder (RQ-VAE) on bar-wise music segments within a Transformer-based encoder-decoder framework.<n>For comprehensive evaluation, we apply MuseTok to music generation and semantic understanding tasks, including melody extraction, chord recognition, and emotion recognition.
  arXiv  Detail & Related papers  (2025-10-18T00:04:48Z)
• Discovering "Words" in Music: Unsupervised Learning of Compositional Sparse Code for Symbolic Music [50.87225308217594]
  This paper presents an unsupervised machine learning algorithm that identifies recurring patterns -- referred to as music-words'' -- from symbolic music data.<n>We formulate the task of music-word discovery as a statistical optimization problem and propose a two-stage Expectation-Maximization (EM)-based learning framework.
  arXiv  Detail & Related papers  (2025-09-29T11:10:57Z)
• Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning [69.78158549955384]
  We introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions.<n>This approach generates verifiable sheet music questions in both textual and visual modalities.<n> Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music.
  arXiv  Detail & Related papers  (2025-09-04T09:42:17Z)
• Large Language Models' Internal Perception of Symbolic Music [3.9901365062418317]
  Large language models (LLMs) excel at modeling relationships between strings in natural language.<n>This paper investigates how LLMs represent musical concepts by generating symbolic music data from textual prompts.
  arXiv  Detail & Related papers  (2025-07-17T05:48:45Z)
• SongComposer: A Large Language Model for Lyric and Melody Generation in Song Composition [82.38021790213752]
  SongComposer is a music-specialized large language model (LLM)<n>It integrates the capability of simultaneously composing melodies into LLMs by leveraging three key innovations.<n>It outperforms advanced LLMs in tasks such as lyric-to-melody generation, melody-to-lyric generation, song continuation, and text-to-song creation.<n>We will release SongCompose, a large-scale dataset for training, containing paired lyrics and melodies in Chinese and English.
  arXiv  Detail & Related papers  (2024-02-27T16:15:28Z)
• MARBLE: Music Audio Representation Benchmark for Universal Evaluation [79.25065218663458]
  We introduce the Music Audio Representation Benchmark for universaL Evaluation, termed MARBLE. It aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining a comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. We then establish a unified protocol based on 14 tasks on 8 public-available datasets, providing a fair and standard assessment of representations of all open-sourced pre-trained models developed on music recordings as baselines.
  arXiv  Detail & Related papers  (2023-06-18T12:56:46Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.