Factual and Musical Evaluation Metrics for Music Language Models

• URL: http://arxiv.org/abs/2511.05550v1
• Date: Sun, 02 Nov 2025 18:08:26 GMT
• Title: Factual and Musical Evaluation Metrics for Music Language Models
• Authors: Daniel Chenyu Lin, Michael Freeman, John Thickstun,
• Abstract summary: Music language models (Music LMs) leverage multimodal representations to answer natural language queries about musical audio recordings.<n>Although Music LMs are reportedly improving, we find that current evaluations fail to capture whether their answers are correct.<n>We propose (1) a better general-purpose evaluation metric for Music LMs adapted to the music domain and (2) a factual evaluation framework to quantify the correctness of a Music LM's responses.
• Score: 5.124350492915739
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Music language models (Music LMs), like vision language models, leverage multimodal representations to answer natural language queries about musical audio recordings. Although Music LMs are reportedly improving, we find that current evaluations fail to capture whether their answers are correct. Specifically, for all Music LMs that we examine, widely-used evaluation metrics such as BLEU, METEOR, and BERTScore fail to measure anything beyond linguistic fluency of the model's responses. To measure the true performance of Music LMs, we propose (1) a better general-purpose evaluation metric for Music LMs adapted to the music domain and (2) a factual evaluation framework to quantify the correctness of a Music LM's responses. Our framework is agnostic to the modality of the question-answering model and could be generalized to quantify performance in other open-ended question-answering domains. We use open datasets in our experiments and will release all code on publication.

Related papers

• Music Flamingo: Scaling Music Understanding in Audio Language Models [98.94537017112704]
  Music Flamingo is a novel large audio-language model designed to advance music understanding in foundational audio models.<n> MF-Skills is a dataset labeled through a multi-stage pipeline that yields rich captions and question-answer pairs covering harmony, structure, timbre, lyrics, and cultural context.<n>We introduce a post-training recipe: we first cold-start with MF-Think, a novel chain-of-thought dataset grounded in music theory, followed by GRPO-based reinforcement learning with custom rewards.
  arXiv  Detail & Related papers  (2025-11-13T13:21:09Z)
• Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning [69.78158549955384]
  We introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions.<n>This approach generates verifiable sheet music questions in both textual and visual modalities.<n> Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music.
  arXiv  Detail & Related papers  (2025-09-04T09:42:17Z)
• CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following [12.638115555721257]
  CMI-Bench is a comprehensive music instruction following benchmark designed to evaluate audio-text LLMs on a diverse set of music information retrieval (MIR) tasks.<n>Unlike previous benchmarks, CMI-Bench adopts standardized evaluation metrics consistent with previous state-of-the-art MIR models.<n>We provide an evaluation toolkit supporting all open-source audio-textual LLMs, including LTU, Qwen-audio, SALMONN, MusiLingo, etc.
  arXiv  Detail & Related papers  (2025-06-14T00:18:44Z)
• Harnessing High-Level Song Descriptors towards Natural Language-Based Music Recommendation [10.740852246735004]
  Language Models (LMs) have gained popularity in assisting users to navigate large catalogs. We assess LMs effectiveness in recommending songs based on user natural language descriptions and items with descriptors like genres, moods, and listening contexts. Our findings reveal improved performance as LMs are fine-tuned for general language similarity, information retrieval, and mapping longer descriptions to shorter, high-level descriptors in music.
  arXiv  Detail & Related papers  (2024-11-08T15:45:33Z)
• Evaluation of pretrained language models on music understanding [0.0]
  We demonstrate that Large Language Models (LLM) suffer from 1) prompt sensitivity, 2) inability to model negation, and 3) sensitivity towards the presence of specific words. We quantified these properties as a triplet-based accuracy, evaluating the ability to model the relative similarity of labels in a hierarchical ontology. Despite the relatively high accuracy reported, inconsistencies are evident in all six models, suggesting that off-the-shelf LLMs need adaptation to music before use.
  arXiv  Detail & Related papers  (2024-09-17T14:44:49Z)
• MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models [11.834712543531756]
  MuChoMusic is a benchmark for evaluating music understanding in multimodal language models focused on audio. It comprises 1,187 multiple-choice questions, all validated by human annotators, on 644 music tracks sourced from two publicly available music datasets. We evaluate five open-source models and identify several pitfalls, including an over-reliance on the language modality.
  arXiv  Detail & Related papers  (2024-08-02T15:34:05Z)
• The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models [63.53530525014976]
  ZIQI-Eval is a benchmark specifically designed to evaluate the music-related capabilities of large language models (LLMs) ZIQI-Eval encompasses a wide range of questions, covering 10 major categories and 56 subcategories, resulting in over 14,000 meticulously curated data entries. Results indicate that all LLMs perform poorly on the ZIQI-Eval benchmark, suggesting significant room for improvement in their musical capabilities.
  arXiv  Detail & Related papers  (2024-06-22T16:24:42Z)
• ChatMusician: Understanding and Generating Music Intrinsically with LLM [81.48629006702409]
  ChatMusician is an open-source Large Language Models (LLMs) that integrates intrinsic musical abilities. It can understand and generate music with a pure text tokenizer without any external multi-modal neural structures or tokenizers. Our model is capable of composing well-structured, full-length music, conditioned on texts, chords, melodies, motifs, musical forms, etc.
  arXiv  Detail & Related papers  (2024-02-25T17:19:41Z)
• MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music [21.380568107727207]
  We present MuChin, the first open-source music description benchmark in Chinese colloquial language. MuChin is designed to evaluate the performance of multimodal Large Language Models in understanding and describing music. All data related to the benchmark, along with the scoring code and detailed appendices, have been open-sourced.
  arXiv  Detail & Related papers  (2024-02-15T10:55:01Z)
• Generating Benchmarks for Factuality Evaluation of Language Models [61.69950787311278]
  We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation.
  arXiv  Detail & Related papers  (2023-07-13T17:14:38Z)
• MARBLE: Music Audio Representation Benchmark for Universal Evaluation [79.25065218663458]
  We introduce the Music Audio Representation Benchmark for universaL Evaluation, termed MARBLE. It aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining a comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. We then establish a unified protocol based on 14 tasks on 8 public-available datasets, providing a fair and standard assessment of representations of all open-sourced pre-trained models developed on music recordings as baselines.
  arXiv  Detail & Related papers  (2023-06-18T12:56:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.