What Are You Listening to? Temporal Music Grounding for Audio-to-Text Large Language Models

• URL: http://arxiv.org/abs/2608.29480v1
• Date: Sun, 30 Aug 2026 00:17:28 GMT
• Title: What Are You Listening to? Temporal Music Grounding for Audio-to-Text Large Language Models
• Abstract summary: We introduce temporal music grounding, a task in which a model returns one or more time spans corresponding to a queried musical note, event, or pattern.<n>We present MusicGroundingBench, a benchmark suite built by rendering algorithmically generated piano MIDI to audio.<n>Experiments show that temporal music grounding remains challenging for current audio-language models, whereas task-specific training yields substantial gains.
• Score: 4.715803044908279
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large audio-language models can produce fluent and musically plausible responses, yet it often remains unclear whether those responses are grounded in the audio input. We introduce temporal music grounding, a task in which a model returns one or more time spans corresponding to a queried musical note, event, or pattern. To evaluate this capability, we present MusicGroundingBench, a controlled benchmark suite built by rendering algorithmically generated piano MIDI to audio, yielding exact symbolic-to-audio alignment. The suite comprises two subsets: MGBench-3N, which evaluates note-level grounding in clips containing up to three notes, and MGBench-2B, which evaluates structured grounding and short-form music understanding in two-bar excerpts. Experiments show that temporal music grounding remains challenging for current audio-language models, whereas task-specific training yields substantial gains. We further report exploratory evidence on the relationship between grounding supervision and music understanding. These results establish MusicGroundingBench as a controlled testbed for assessing whether audio-language models ground their responses in temporally localized musical evidence.

Related papers

• Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models [51.75139461584678]
  Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order.<n>We propose an instruction-level framework that uses audio-aware large language models as fine-grained judges to verify target event presence and temporal relations.<n>Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.
  arXiv  Detail & Related papers  (2026-07-15T03:13:06Z)
• Continuous Audio Thinking for Large Audio Language Models [16.335310406868217]
  We introduce Continuous Audio Thinking (CoAT) to equip audio language models with a continuous latent workspace for organizing acoustic information prior to response generation.<n>Within the thinking space, the model can utilize the rich acoustic information provided by expert distillation when generating its response.<n>CoAT does not require additional autoregressive decoding cost over the baseline.
  arXiv  Detail & Related papers  (2026-06-05T11:38:30Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• PitchBench: Measuring Pitch Hearing in Audio-Language Models [9.085153261594623]
  We introduce PitchBench, an evaluation suite that measures pitch hearing in audio-language models (ALMs)<n>PitchBench comprises 28 experiments spanning absolute and relative pitch perception within sequences and chords, while varying loudness, note duration, sound source, time stretching, background noise, and other acoustic conditions.<n>We find that pitch hearing remains highly unreliable, with accuracy varying sharply by sound source, note duration, and notation format.
  arXiv  Detail & Related papers  (2026-05-25T05:37:15Z)
• Rubato: Transcribing Piano Music with Timestamps [47.76733897717644]
  We consider the conversion of musical recordings into human-readable sheet music annotated with timestamps.<n>We introduce (1) a prompt-conditioned encoder-decoder model, named Rubato, trained to output (2) a new textual representation for polyphonic music, named InterMo.<n>Our experiments demonstrate that Rubato produces timestamped piano sheet music from audio with higher notational accuracy than the best existing approaches.
  arXiv  Detail & Related papers  (2026-05-22T23:47:21Z)
• Music Flamingo: Scaling Music Understanding in Audio Language Models [98.94537017112704]
  Music Flamingo is a novel large audio-language model designed to advance music understanding in foundational audio models.<n> MF-Skills is a dataset labeled through a multi-stage pipeline that yields rich captions and question-answer pairs covering harmony, structure, timbre, lyrics, and cultural context.<n>We introduce a post-training recipe: we first cold-start with MF-Think, a novel chain-of-thought dataset grounded in music theory, followed by GRPO-based reinforcement learning with custom rewards.
  arXiv  Detail & Related papers  (2025-11-13T13:21:09Z)
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs [53.248502396225724]
  AudioMarathon is a benchmark designed to evaluate both understanding and inference efficiency on long-form audio.<n>We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows.<n>The results show large gaps across current LALMs and highlight the need for better temporal reasoning.
  arXiv  Detail & Related papers  (2025-10-08T17:50:16Z)
• Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning [69.78158549955384]
  We introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions.<n>This approach generates verifiable sheet music questions in both textual and visual modalities.<n> Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music.
  arXiv  Detail & Related papers  (2025-09-04T09:42:17Z)
• CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation [17.41880273107978]
  Contrastive Long-form Language-Audio Pretraining (textbfCoLLAP) We propose Contrastive Long-form Language-Audio Pretraining (textbfCoLLAP) to significantly extend the perception window for both the input audio (up to 5 minutes) and the language descriptions (exceeding 250 words) We collect 51.3K audio-text pairs derived from the large-scale AudioSet training dataset, where the average audio length reaches 288 seconds.
  arXiv  Detail & Related papers  (2024-10-03T07:46:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.