MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models

• URL: http://arxiv.org/abs/2608.22236v2
• Date: Mon, 31 Aug 2026 23:51:41 GMT
• Title: MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models
• Abstract summary: We introduce MRMAD, a benchmark for evaluating audio degradation perception and understanding in large audio-language models (LALMs)<n> MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues across multiple audio inputs.<n>We find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably.
• Score: 11.14810883033264
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues across multiple audio inputs, requiring models to identify types of degradation, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and comprehend low-level acoustic phenomena over multi-turn dialogues. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. Human evaluations further reveal a significant perception gap between LALMs and human listeners. MRMAD thus exposes a critical yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.

Related papers

• MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation [56.428467142804756]
  We introduce MTAVG-Bench 2.0, a benchmark for diagnosing failure modes of cinematic expressiveness in multi-talker audio-video generation.<n>MTAVG-Bench 2.0 targets short-drama and scene-level generation, and establishes a high-level failure taxonomy spanning acting, narrative, atmosphere, and audio-visual language.<n> Experimental results show that commercial omni models such as Gemini substantially outperform other evaluators.
  arXiv  Detail & Related papers  (2026-05-27T06:38:54Z)
• HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models [30.18524844766061]
  We introduce HalluAudio, the first large-scale benchmark for evaluating hallucinations across speech, environmental sound, and music.<n>HalluAudio comprises over 5K human-verified QA pairs and spans diverse task types, including binary judgments, multi-choice reasoning, attribute verification, and open-ended QA.<n>Our results reveal significant deficiencies in acoustic grounding, temporal reasoning, and music attribute understanding, underscoring the need for reliable and robust LALMs.
  arXiv  Detail & Related papers  (2026-04-21T10:05:28Z)
• Do Audio-Visual Large Language Models Really See and Hear? [66.8531220331223]
  We analyze how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs.<n>Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.
  arXiv  Detail & Related papers  (2026-04-03T00:48:49Z)
• How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation [97.0235251827591]
  Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs)<n>We study how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance.<n>Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance.
  arXiv  Detail & Related papers  (2026-03-19T17:50:07Z)
• SpeakerSleuth: Evaluating Large Audio-Language Models as Judges for Multi-turn Speaker Consistency [12.420484491347073]
  We present SpeakerSleuth, a benchmark evaluating whether LALMs can reliably judge speaker consistency in multi-turn dialogues.<n>We construct 1,818 human-verified evaluation instances across four diverse datasets spanning synthetic and real speech.<n>We find that models struggle to reliably detect acoustic inconsistencies.
  arXiv  Detail & Related papers  (2026-01-07T15:45:41Z)
• Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models [49.097347801692166]
  We introduce Thinking-with-Sound (TwS), a framework that equips Large Audio-Language Models with Audio CoT.<n>TwS enables models to actively think with audio signals, performing numerical analysis and digital manipulation through multimodal reasoning.<n>Experiments reveal that state-of-the-art LALMs suffer dramatic performance degradation on MELD-Hard1k, with accuracy dropping by more than $50%$ compared to clean audio.
  arXiv  Detail & Related papers  (2025-09-26T01:27:59Z)
• WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations [67.6147632074449]
  We introduce the World-of-Whale benchmark (WoW-Bench) to evaluate low-level auditory perception and cognition using marine mammal vocalizations.<n>WoW-Bench is composed of a Perception benchmark for categorizing novel sounds and a Cognition benchmark, inspired by Bloom's taxonomy, to assess the abilities to remember, understand, apply, and analyze sound events.<n>Experiments with state-of-the-art LALMs show performance far below human levels, indicating a need for stronger auditory grounding in LALMs.
  arXiv  Detail & Related papers  (2025-08-28T16:29:46Z)
• When Audio and Text Disagree: Revealing Text Bias in Large Audio-Language Models [18.160420407067743]
  MCR-BENCH is the first benchmark designed to evaluate how LALMs prioritize information when presented with inconsistent audio-text pairs.<n>We reveal a concerning phenomenon: when inconsistencies exist between modalities, LALMs display a significant bias toward textual input.<n>This tendency leads to substantial performance degradation in audio-centric tasks and raises important reliability concerns for real-world applications.
  arXiv  Detail & Related papers  (2025-08-21T09:58:24Z)
• Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models [49.87432626548563]
  We introduce methods to assess the extent of object hallucination of publicly available LALMs. Our findings reveal that LALMs are comparable to specialized audio captioning models in their understanding of audio content. We explore the potential of prompt engineering to enhance LALMs' performance on discriminative questions.
  arXiv  Detail & Related papers  (2024-06-12T16:51:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.