AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA

• URL: http://arxiv.org/abs/2604.21766v1
• Date: Thu, 23 Apr 2026 15:22:36 GMT
• Title: AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA
• Abstract summary: AUDITA is a large-scale, real-world benchmark to rigorously evaluate audio reasoning beyond surface-level acoustic recognition.<n>Human average accuracy of 32.13% shows both the challenge of the task while demonstrating meaningful comprehension of the audio.<n>State-the-art audio question answering models perform poorly, with average accuracy below 8.86%.
• Score: 7.0864879068510005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing audio question answering benchmarks largely emphasize sound event classification or caption-grounded queries, often enabling models to succeed through shortcut strategies, short-duration cues, lexical priors, dataset-specific biases, or even bypassing audio via metadata and captions rather than genuine reasoning Thus, we present AUDITA (Audio Understanding from Diverse Internet Trivia Authors), a large-scale, real-world benchmark to rigorously evaluate audio reasoning beyond surface-level acoustic recognition. AUDITA comprises carefully curated, human-authored trivia questions grounded in real-world audio, designed to stress robust auditory reasoning through challenging distractors and long-range temporal dependencies, using probing queries that cannot be answered from isolated text or sound cues alone. Human average accuracy of 32.13% shows both the challenge of the task while demonstrating meaningful comprehension of the audio. In stark contrast, state of-the-art audio question answering models perform poorly, with average accuracy below 8.86%. Beyond raw accuracy, we apply Item Response Theory (IRT) to estimate latent proficiency, question difficulty, and expose systematic deficiencies of the models and data.

Related papers

• A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Models [64.68800440221138]
  Large Audio Language Models (LALMs) achieve strong performance on a variety of audio understanding tasks but struggle with temporal reasoning.<n>We introduce a benchmark with 1,657 questions across three foundational tasks designed for mechanistic analysis.<n>Comparing attention upweighting against scaling, we find that redistributing attention across audio tokens is more effective than increasing audio attention.
  arXiv  Detail & Related papers  (2026-06-16T01:57:56Z)
• ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval [9.400944614656735]
  ReasonAudio is the first reasoning-intensive benchmark for Text-Audio Retrieval.<n>It comprises 1,000 queries and 10,000 composite audio clips across five fundamental reasoning tasks: Negation, Order, Overlap, Duration, and Mix.<n>Our evaluation of ten state-of-the-art models reveals the following findings: All models struggle with reasoning-intensive audio retrieval.
  arXiv  Detail & Related papers  (2026-05-05T04:44:51Z)
• All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation [45.45465533352999]
  Large Audio-Language Models show consistent performance gains across speech and audio benchmarks, yet high scores may not reflect true auditory perception.<n>We present a diagnostic framework using two axes: text prior, which measures answerability from text and general knowledge alone, and audio reliance, which assesses actual dependency on the acoustic signal.
  arXiv  Detail & Related papers  (2026-04-27T12:25:18Z)
• HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models [30.18524844766061]
  We introduce HalluAudio, the first large-scale benchmark for evaluating hallucinations across speech, environmental sound, and music.<n>HalluAudio comprises over 5K human-verified QA pairs and spans diverse task types, including binary judgments, multi-choice reasoning, attribute verification, and open-ended QA.<n>Our results reveal significant deficiencies in acoustic grounding, temporal reasoning, and music attribute understanding, underscoring the need for reliable and robust LALMs.
  arXiv  Detail & Related papers  (2026-04-21T10:05:28Z)
• HumMusQA: A Human-written Music Understanding QA Benchmark Dataset [2.7761207021407217]
  This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training.<n>To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.
  arXiv  Detail & Related papers  (2026-03-29T21:33:07Z)
• AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering [58.04745279785462]
  AQUA-Bench is a benchmark for Audio Question Unanswerability Assessment.<n>It systematically evaluates three scenarios: Absent Answer Detection, Incompatible Answer Set Detection, and Incompatible Audio Question Detection.<n>By assessing these cases, AQUA-Bench offers a rigorous measure of model reliability.
  arXiv  Detail & Related papers  (2026-01-18T03:55:28Z)
• MDAR: A Multi-scene Dynamic Audio Reasoning Benchmark [64.89810922949984]
  We introduce MDAR, a benchmark for evaluating models on complex, multi-scene, and dynamically evolving audio reasoning tasks.<n>MDAR comprises 3,000 carefully curated question-answer pairs linked to diverse audio clips.<n>We benchmark 26 state-of-the-art audio language models on MDAR and observe that they exhibit limitations in complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-09-26T15:12:46Z)
• WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations [67.6147632074449]
  We introduce the World-of-Whale benchmark (WoW-Bench) to evaluate low-level auditory perception and cognition using marine mammal vocalizations.<n>WoW-Bench is composed of a Perception benchmark for categorizing novel sounds and a Cognition benchmark, inspired by Bloom's taxonomy, to assess the abilities to remember, understand, apply, and analyze sound events.<n>Experiments with state-of-the-art LALMs show performance far below human levels, indicating a need for stronger auditory grounding in LALMs.
  arXiv  Detail & Related papers  (2025-08-28T16:29:46Z)
• Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge [102.84031769492708]
  This task defines three QA subsets to test audio-language models on interactive question-answering over diverse acoustic scenes.<n>Preliminary results on the development set are compared, showing strong variation across models and subsets.<n>This challenge aims to advance the audio understanding and reasoning capabilities of audio-language models toward human-level acuity.
  arXiv  Detail & Related papers  (2025-05-12T09:04:16Z)
• Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning [55.2480439325792]
  Large audio-language models (LALMs) have shown impressive capabilities in understanding and reasoning about audio and speech information.<n>These models still face challenges, including hallucinating non-existent sound events, misidentifying the order of sound events, and incorrectly attributing sound sources.
  arXiv  Detail & Related papers  (2024-10-21T15:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.