Exploring GPT's Ability as a Judge in Music Understanding

• URL: http://arxiv.org/abs/2501.13261v1
• Date: Wed, 22 Jan 2025 22:49:27 GMT
• Title: Exploring GPT's Ability as a Judge in Music Understanding
• Authors: Kun Fang, Ziyu Wang, Gus Xia, Ichiro Fujinaga,
• Abstract summary: We use a systematic prompt engineering approach for Large Language Models to solve music information retrieval problems.<n>We convert the music data to symbolic inputs and evaluate LLMs' ability in detecting annotation errors in three key MIR tasks.<n>A concept augmentation method is proposed to evaluate LLMs' music reasoning consistency with the provided music concepts in the prompts.
• Score: 6.5178028874627705
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress in text-based Large Language Models (LLMs) and their extended ability to process multi-modal sensory data have led us to explore their applicability in addressing music information retrieval (MIR) challenges. In this paper, we use a systematic prompt engineering approach for LLMs to solve MIR problems. We convert the music data to symbolic inputs and evaluate LLMs' ability in detecting annotation errors in three key MIR tasks: beat tracking, chord extraction, and key estimation. A concept augmentation method is proposed to evaluate LLMs' music reasoning consistency with the provided music concepts in the prompts. Our experiments tested the MIR capabilities of Generative Pre-trained Transformers (GPT). Results show that GPT has an error detection accuracy of 65.20%, 64.80%, and 59.72% in beat tracking, chord extraction, and key estimation tasks, respectively, all exceeding the random baseline. Moreover, we observe a positive correlation between GPT's error finding accuracy and the amount of concept information provided. The current findings based on symbolic music input provide a solid ground for future LLM-based MIR research.

Related papers

• CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following [12.638115555721257]
  CMI-Bench is a comprehensive music instruction following benchmark designed to evaluate audio-text LLMs on a diverse set of music information retrieval (MIR) tasks.<n>Unlike previous benchmarks, CMI-Bench adopts standardized evaluation metrics consistent with previous state-of-the-art MIR models.<n>We provide an evaluation toolkit supporting all open-source audio-textual LLMs, including LTU, Qwen-audio, SALMONN, MusiLingo, etc.
  arXiv  Detail & Related papers  (2025-06-14T00:18:44Z)
• Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers [74.17516978246152]
  Large language models (LLMs) have been widely integrated into information retrieval to advance traditional techniques.<n>We propose EXSEARCH, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds.<n>Experiments on four knowledge-intensive benchmarks show that EXSEARCH substantially outperforms baselines.
  arXiv  Detail & Related papers  (2025-05-26T15:27:55Z)
• GainRAG: Preference Alignment in Retrieval-Augmented Generation through Gain Signal Synthesis [30.185213495829164]
  The Retrieval-Augmented Generation (RAG) framework introduces a retrieval module to dynamically inject retrieved information into the input context of large language models (LLMs)<n>We propose GainRAG, a novel approach that aligns the retriever's and LLM's preferences by defining a new metric, "gain", which measure how well an input passage contributes to correct outputs.<n>The experimental results on 6 datasets verify the effectiveness of GainRAG.
  arXiv  Detail & Related papers  (2025-05-24T14:14:57Z)
• A Mamba-based Network for Semi-supervised Singing Melody Extraction Using Confidence Binary Regularization [14.501400507234356]
  Singing melody extraction is a key task in the field of music information retrieval.<n>Existing methods are facing several limitations.<n>We propose a mamba-based network, called SpectMamba, for semi-supervised singing melody extraction.
  arXiv  Detail & Related papers  (2025-05-13T15:43:35Z)
• Grounded Chain-of-Thought for Multimodal Large Language Models [66.04061083611863]
  We propose a new learning task for multimodal large language models (MLLMs) called Grounded Chain-of-Thought (GCoT) GCoT is keen to helping MLLMs to recognize and ground the relevant visual cues step by step, thereby predicting the correct answer with grounding coordinates as the intuitive basis. To facilitate this task, we also carefully design and construct a dataset called multimodal grounded chain-of-thought (MM-GCoT) consisting of 24,022 GCoT examples for 5,033 images.
  arXiv  Detail & Related papers  (2025-03-17T04:07:47Z)
• Invar-RAG: Invariant LLM-aligned Retrieval for Better Generation [43.630437906898635]
  We propose a novel two-stage fine-tuning architecture called Invar-RAG. In the retrieval stage, an LLM-based retriever is constructed by integrating LoRA-based representation learning. In the generation stage, a refined fine-tuning method is employed to improve LLM accuracy in generating answers based on retrieved information.
  arXiv  Detail & Related papers  (2024-11-11T14:25:37Z)
• Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement [51.601916604301685]
  Large language models (LLMs) generate content that can undermine trust in online discourse. Current methods often focus on binary classification, failing to address the complexities of real-world scenarios like human-AI collaboration. To move beyond binary classification and address these challenges, we propose a new paradigm for detecting LLM-generated content.
  arXiv  Detail & Related papers  (2024-10-18T08:14:10Z)
• Diver: Large Language Model Decoding with Span-Level Mutual Information Verification [13.378881059577635]
  Diver is a novel approach that enhances LLM Decoding through span-level PMI verification. We evaluate our method across various downstream tasks, and empirical results demonstrate that Diver significantly outperforms existing decoding methods in both performance and versatility.
  arXiv  Detail & Related papers  (2024-06-04T09:02:22Z)
• FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answering [46.41364317172677]
  Large language models (LLMs) are often challenged by generating erroneous or hallucinated responses. We propose a unified framework, FiDeLiS, designed to improve the factuality of LLM responses by anchoring answers to verifiable reasoning steps retrieved from a KG.
  arXiv  Detail & Related papers  (2024-05-22T17:56:53Z)
• HARGPT: Are LLMs Zero-Shot Human Activity Recognizers? [9.414529772034985]
  We show that Large Language Models (LLMs) can comprehend raw IMU data and perform human activity recognition tasks in a zero-shot manner. We benchmark HARGPT on GPT4 using two public datasets of different inter-class similarities and compare various baselines both based on traditional machine learning and state-of-the-art deep classification models. Remarkably, LLMs successfully recognize human activities from raw IMU data and consistently outperform all the baselines on both datasets.
  arXiv  Detail & Related papers  (2024-03-05T07:34:51Z)
• MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline [12.186691561822256]
  We postulate that the inherent nature of large language models (LLMs) presents challenges in modeling mathematical reasoning. This paper introduces a novel math dataset, enhanced with a capability to utilize a Python code interpreter. We propose a tentative, easily replicable protocol for the fine-tuning of math-specific LLMs.
  arXiv  Detail & Related papers  (2024-01-16T08:08:01Z)
• A Survey on Detection of LLMs-Generated Content [97.87912800179531]
  The ability to detect LLMs-generated content has become of paramount importance. We aim to provide a detailed overview of existing detection strategies and benchmarks. We also posit the necessity for a multi-faceted approach to defend against various attacks.
  arXiv  Detail & Related papers  (2023-10-24T09:10:26Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs. We introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
  arXiv  Detail & Related papers  (2023-10-10T16:38:49Z)
• Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks [121.74957524305283]
  This paper proposes a novel framework named textbfSearch-in-the-Chain (SearChain) for the interaction between Information Retrieval (IR) and Large Language Model (LLM) Experiments show that SearChain outperforms state-of-the-art baselines on complex knowledge-intensive tasks.
  arXiv  Detail & Related papers  (2023-04-28T10:15:25Z)
• Robust Meta-learning with Sampling Noise and Label Noise via Eigen-Reptile [78.1212767880785]
  meta-learner is prone to overfitting since there are only a few available samples. When handling the data with noisy labels, the meta-learner could be extremely sensitive to label noise. We present Eigen-Reptile (ER) that updates the meta- parameters with the main direction of historical task-specific parameters.
  arXiv  Detail & Related papers  (2022-06-04T08:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.