Music Flamingo: Scaling Music Understanding in Audio Language Models

• URL: http://arxiv.org/abs/2511.10289v1
• Date: Fri, 14 Nov 2025 01:43:47 GMT
• Title: Music Flamingo: Scaling Music Understanding in Audio Language Models
• Authors: Sreyan Ghosh, Arushi Goel, Lasha Koroshinadze, Sang-gil Lee, Zhifeng Kong, Joao Felipe Santos, Ramani Duraiswami, Dinesh Manocha, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro,
• Abstract summary: Music Flamingo is a novel large audio-language model designed to advance music understanding in foundational audio models.<n> MF-Skills is a dataset labeled through a multi-stage pipeline that yields rich captions and question-answer pairs covering harmony, structure, timbre, lyrics, and cultural context.<n>We introduce a post-training recipe: we first cold-start with MF-Think, a novel chain-of-thought dataset grounded in music theory, followed by GRPO-based reinforcement learning with custom rewards.
• Score: 98.94537017112704
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Music Flamingo, a novel large audio-language model designed to advance music (including song) understanding in foundational audio models. While audio-language research has progressed rapidly, music remains challenging due to its dynamic, layered, and information-dense nature. Progress has been further limited by the difficulty of scaling open audio understanding models, primarily because of the scarcity of high-quality music data and annotations. As a result, prior models are restricted to producing short, high-level captions, answering only surface-level questions, and showing limited generalization across diverse musical cultures. To address these challenges, we curate MF-Skills, a large-scale dataset labeled through a multi-stage pipeline that yields rich captions and question-answer pairs covering harmony, structure, timbre, lyrics, and cultural context. We fine-tune an enhanced Audio Flamingo 3 backbone on MF-Skills and further strengthen multiple skills relevant to music understanding. To improve the model's reasoning abilities, we introduce a post-training recipe: we first cold-start with MF-Think, a novel chain-of-thought dataset grounded in music theory, followed by GRPO-based reinforcement learning with custom rewards. Music Flamingo achieves state-of-the-art results across 10+ benchmarks for music understanding and reasoning, establishing itself as a generalist and musically intelligent audio-language model. Beyond strong empirical results, Music Flamingo sets a new standard for advanced music understanding by demonstrating how models can move from surface-level recognition toward layered, human-like perception of songs. We believe this work provides both a benchmark and a foundation for the community to build the next generation of models that engage with music as meaningfully as humans do.

Related papers

• Advancing the Foundation Model for Music Understanding [9.210248657997687]
  We introduce a unified foundation model named MuFun for holistic music understanding.<n>Our model features a novel architecture that jointly processes instrumental and lyrical content.<n>We also propose a new benchmark for multi-faceted music understanding called MuCUE.
  arXiv  Detail & Related papers  (2025-08-02T03:33:47Z)
• From Generality to Mastery: Composer-Style Symbolic Music Generation via Large-Scale Pre-training [4.7205815347741185]
  We investigate how general music knowledge learned from a broad corpus can enhance the mastery of specific composer styles.<n>First, we pre-train a REMI-based music generation model on a large corpus of pop, folk, and classical music.<n>Then, we fine-tune it on a small, human-verified dataset from four renowned composers, namely Bach, Mozart, Beethoven, and Chopin.
  arXiv  Detail & Related papers  (2025-06-20T22:20:59Z)
• Learning Musical Representations for Music Performance Question Answering [10.912207282129753]
  multimodal learning methods are incapable of dealing with fundamental problems within the music performances.<n>Our primary backbone is designed to incorporate multimodal interactions within the context of music data.<n>Our experiments show state-of-the-art effects on the Music AVQA datasets.
  arXiv  Detail & Related papers  (2025-02-10T17:41:57Z)
• A Survey of Foundation Models for Music Understanding [60.83532699497597]
  This work is one of the early reviews of the intersection of AI techniques and music understanding. We investigated, analyzed, and tested recent large-scale music foundation models in respect of their music comprehension abilities.
  arXiv  Detail & Related papers  (2024-09-15T03:34:14Z)
• Foundation Models for Music: A Survey [77.77088584651268]
  Foundations models (FMs) have profoundly impacted diverse sectors, including music. This comprehensive review examines state-of-the-art (SOTA) pre-trained models and foundation models in music.
  arXiv  Detail & Related papers  (2024-08-26T15:13:14Z)
• MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models [57.47799823804519]
  We are inspired by how musicians compose music not just from a movie script, but also through visualizations. We propose MeLFusion, a model that can effectively use cues from a textual description and the corresponding image to synthesize music. Our exhaustive experimental evaluation suggests that adding visual information to the music synthesis pipeline significantly improves the quality of generated music.
  arXiv  Detail & Related papers  (2024-06-07T06:38:59Z)
• MusicRL: Aligning Music Generation to Human Preferences [62.44903326718772]
  MusicRL is the first music generation system finetuned from human feedback. We deploy MusicLM to users and collect a substantial dataset comprising 300,000 pairwise preferences. We train MusicRL-U, the first text-to-music model that incorporates human feedback at scale.
  arXiv  Detail & Related papers  (2024-02-06T18:36:52Z)
• LLark: A Multimodal Instruction-Following Language Model for Music [7.7033394687966865]
  Music has a unique and complex structure which is challenging for both expert humans and existing AI systems to understand. We present LLark, an instruction-tuned multimodal model for emphmusic understanding.
  arXiv  Detail & Related papers  (2023-10-11T03:12:47Z)
• MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response [42.73982391253872]
  MusiLingo is a novel system for music caption generation and music-related query responses. We train it on an extensive music caption dataset and fine-tune it with instructional data. Empirical evaluations demonstrate its competitive performance in generating music captions and composing music-related Q&A pairs.
  arXiv  Detail & Related papers  (2023-09-15T19:31:40Z)
• MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training [97.91071692716406]
  Symbolic music understanding refers to the understanding of music from the symbolic data. MusicBERT is a large-scale pre-trained model for music understanding.
  arXiv  Detail & Related papers  (2021-06-10T10:13:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.