Investigating Modality Contribution in Audio LLMs for Music

• URL: http://arxiv.org/abs/2509.20641v1
• Date: Thu, 25 Sep 2025 00:56:35 GMT
• Title: Investigating Modality Contribution in Audio LLMs for Music
• Authors: Giovana Morais, Magdalena Fuentes,
• Abstract summary: Audio Large Language Models (Audio LLMs) enable human-like conversation about music.<n>It is unclear if they are truly listening to the audio or just using textual reasoning.<n>This paper investigates this issue by quantifying the contribution of each modality to a model's output.
• Score: 8.118262908070152
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio Large Language Models (Audio LLMs) enable human-like conversation about music, yet it is unclear if they are truly listening to the audio or just using textual reasoning, as recent benchmarks suggest. This paper investigates this issue by quantifying the contribution of each modality to a model's output. We adapt the MM-SHAP framework, a performance-agnostic score based on Shapley values that quantifies the relative contribution of each modality to a model's prediction. We evaluate two models on the MuChoMusic benchmark and find that the model with higher accuracy relies more on text to answer questions, but further inspection shows that even if the overall audio contribution is low, models can successfully localize key sound events, suggesting that audio is not entirely ignored. Our study is the first application of MM-SHAP to Audio LLMs and we hope it will serve as a foundational step for future research in explainable AI and audio.

Related papers

• UALM: Unified Audio Language Model for Understanding, Generation and Reasoning [124.19449187588832]
  Unified Audio Language Model (UALM) aims to unify audio understanding, text-to-audio generation, and multimodal reasoning in a single model.<n>We first present UALM-Gen, a text-to-audio language model that directly predicts audio tokens and is comparable to state-of-the-art diffusion-based models.<n>We present UALM-Reason, a multimodal reasoning model that utilizes both text and audio in the intermediate thinking steps to facilitate complex generation tasks.
  arXiv  Detail & Related papers  (2025-10-13T22:55:01Z)
• MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation [34.79792511587843]
  We introduce MMAudioSep, a generative model for video/text-queried sound separation.<n>By leveraging knowledge about the relationship between video/text and audio learned through a pretrained audio generative model, we can train the model more efficiently.
  arXiv  Detail & Related papers  (2025-10-10T07:13:06Z)
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs [53.248502396225724]
  AudioMarathon is a benchmark designed to evaluate both understanding and inference efficiency on long-form audio.<n>We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows.<n>The results show large gaps across current LALMs and highlight the need for better temporal reasoning.
  arXiv  Detail & Related papers  (2025-10-08T17:50:16Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval [33.114796739109075]
  Video Moment Retrieval (VMR) aims to retrieve a specific moment semantically related to a given query.<n>Most existing VMR methods solely focus on the visual and textual modalities while neglecting the complementary but important audio modality.<n>We propose a novel Importance-aware Multi-Granularity fusion model (IMG), which learns to dynamically and selectively aggregate the audio-vision-text contexts for VMR.
  arXiv  Detail & Related papers  (2025-08-06T09:58:43Z)
• SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models [43.46082014842855]
  We introduce SoundMind, a dataset of 6,446 audio-text annotated samples specifically curated to support complex reasoning.<n>We then propose SoundMind-RL, a rule-based reinforcement learning (RL) algorithm designed to equip audio-language models with robust audio-text reasoning capabilities.<n>This work highlights the benefit of combining high-quality, reasoning-focused datasets with specialized RL techniques, and contributes to advancing auditory intelligence in language models.
  arXiv  Detail & Related papers  (2025-06-15T18:26:08Z)
• AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? [65.49972312524724]
  multimodal large language models (MLLMs) have expanded their capabilities to include vision and audio modalities.<n>Our proposed DeafTest reveals that MLLMs often struggle with simple tasks humans find trivial.<n>We introduce AV-Odyssey Bench, a comprehensive audio-visual benchmark designed to assess whether those MLLMs can truly understand the audio-visual information.
  arXiv  Detail & Related papers  (2024-12-03T17:41:23Z)
• AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension [95.8442896569132]
  We introduce AIR-Bench, the first benchmark to evaluate the ability of Large Audio-Language Models (LALMs) to understand various types of audio signals and interact with humans in the textual format. Results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation.
  arXiv  Detail & Related papers  (2024-02-12T15:41:22Z)
• Retrieval-Augmented Text-to-Audio Generation [36.328134891428085]
  We show that the state-of-the-art models, such as AudioLDM, are biased in their generation performance. We propose a simple retrieval-augmented approach for TTA models. We show that Re-AudioLDM can generate realistic audio for complex scenes, rare audio classes, and even unseen audio types.
  arXiv  Detail & Related papers  (2023-09-14T22:35:39Z)
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models [65.18102159618631]
  multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. We propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps.
  arXiv  Detail & Related papers  (2023-01-30T04:44:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.