ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval

• URL: http://arxiv.org/abs/2605.03361v2
• Date: Wed, 06 May 2026 14:04:55 GMT
• Title: ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval
• Abstract summary: ReasonAudio is the first reasoning-intensive benchmark for Text-Audio Retrieval.<n>It comprises 1,000 queries and 10,000 composite audio clips across five fundamental reasoning tasks: Negation, Order, Overlap, Duration, and Mix.<n>Our evaluation of ten state-of-the-art models reveals the following findings: All models struggle with reasoning-intensive audio retrieval.
• Score: 9.400944614656735
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: As multimodal content continues to expand at a rapid pace, audio retrieval has emerged as a key enabling technology for media search, content organization, and intelligent assistants. However, most existing benchmarks concentrate on semantic matching and fail to capture the fact that real-world queries often demand advanced reasoning abilities, including negation understanding, temporal ordering, concurrent event recognition, and duration discrimination. To address this gap, we introduce ReasonAudio, the first reasoning-intensive benchmark for Text-Audio Retrieval, comprising 1,000 queries and 10,000 composite audio clips across five fundamental reasoning tasks: Negation, Order, Overlap, Duration, and Mix. Despite their intuitive nature for humans and straightforward construction, these tasks pose significant challenges to current models. Our evaluation of ten state-of-the-art models reveals the following findings: All models struggle with reasoning-intensive audio retrieval, performing particularly poorly on Negation and Duration while showing relatively better results on Overlap and Order. Moreover, Multimodal Large Language Model-based embedding models fail to inherit the reasoning capabilities of their backbones through contrastive fine-tuning, suggesting that current training paradigms are insufficient to preserve reasoning capacity in retrieval settings

Related papers

• A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Models [64.68800440221138]
  Large Audio Language Models (LALMs) achieve strong performance on a variety of audio understanding tasks but struggle with temporal reasoning.<n>We introduce a benchmark with 1,657 questions across three foundational tasks designed for mechanistic analysis.<n>Comparing attention upweighting against scaling, we find that redistributing attention across audio tokens is more effective than increasing audio attention.
  arXiv  Detail & Related papers  (2026-06-16T01:57:56Z)
• AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA [7.0864879068510005]
  AUDITA is a large-scale, real-world benchmark to rigorously evaluate audio reasoning beyond surface-level acoustic recognition.<n>Human average accuracy of 32.13% shows both the challenge of the task while demonstrating meaningful comprehension of the audio.<n>State-the-art audio question answering models perform poorly, with average accuracy below 8.86%.
  arXiv  Detail & Related papers  (2026-04-23T15:22:36Z)
• Koopman Regularized Deep Speech Disentanglement for Speaker Verification [6.659299099827954]
  Deep Koopman Speech Disentanglement Autoencoder (DKSD-AE)<n>We propose a structured autoencoder that combines a novel multi-step Koopman operator learning module with instance normalization to disentangle speaker and content dynamics.<n>Our findings suggest that Koopman-based temporal modelling, when combined with instance normalization, provides an efficient and principled solution for speaker-focused representation learning.
  arXiv  Detail & Related papers  (2026-03-05T17:30:18Z)
• Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models [56.851611990473174]
  Reasoning over dynamic visual content remains a central challenge for large language models.<n>We propose a reinforcement learning approach that enhances both temporal precision and reasoning consistency.<n>The resulting model, Video R2, achieves consistently higher TAC, VAS, and accuracy across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-11-28T18:59:58Z)
• Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models [80.75260664100644]
  Mini-Omni-Reasoner is a framework that enables reasoning within speech via a novel "Thinking-in-Speaking" formulation.<n>It interleaves silent reasoning tokens with spoken response tokens at the token level.<n>It achieves a +19.1% gain in arithmetic reasoning and +6.4% in contextual understanding, with shorter outputs and zero decoding latency.
  arXiv  Detail & Related papers  (2025-08-18T15:14:04Z)
• AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning [3.949628618389608]
  AURA is a benchmark for evaluating the cross-modal reasoning capabilities of Audio-Visual Large Language Models (AV-LLMs) and Omni-modal Language Models (OLMs)<n>AURA includes questions across six challenging cognitive domains, such as causality, timbre and pitch, tempo and AV synchronization, unanswerability, implicit distractions, and skill profiling.<n>We propose a novel metric, AuraScore, which addresses the lack of robust tools for evaluating reasoning fidelity.
  arXiv  Detail & Related papers  (2025-08-10T20:06:42Z)
• Discrete Audio Tokens: More Than a Survey! [137.3721175670642]
  This paper presents a systematic review and benchmark of discrete audio tokenizers.<n>It covers speech, music, and general audio domains.<n>We propose a taxonomy of tokenization approaches based on encoder-decoder, quantization techniques, training paradigm, streamability, and application domains.
  arXiv  Detail & Related papers  (2025-06-12T01:35:43Z)
• VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos [89.39873803375498]
  VideoMathQA is a benchmark designed to evaluate whether models can perform temporally extended cross-modal reasoning on videos.<n>The benchmark spans 10 diverse mathematical domains, covering videos ranging from 10 seconds to over 1 hour.<n>It requires models to interpret structured visual content, understand instructional narratives, and jointly ground concepts across visual, audio, and textual modalities.
  arXiv  Detail & Related papers  (2025-06-05T17:59:58Z)
• Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning [55.2480439325792]
  Large audio-language models (LALMs) have shown impressive capabilities in understanding and reasoning about audio and speech information.<n>These models still face challenges, including hallucinating non-existent sound events, misidentifying the order of sound events, and incorrectly attributing sound sources.
  arXiv  Detail & Related papers  (2024-10-21T15:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.