ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions

• URL: http://arxiv.org/abs/2607.17812v1
• Date: Mon, 20 Jul 2026 10:59:15 GMT
• Title: ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions
• Abstract summary: ESCUCHA comprises 1,000 human-curated questions paired with audio, totaling 162.9 hours sourced directly from the wild'' rather than drawn from existing datasets.<n>The benchmark emphasizes reasoning, spanning 9 perceptual and 10 reasoning categories, and it captures linguistic diversity through multiple Spanish accents and non-normative speech.<n> Benchmarking several state-of-the-art multimodal and speech models reveals substantial performance gaps relative to trained humans.
• Score: 66.93528077633728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large audio language models (LALMs) advance, robust evaluation frameworks have become essential. In this context, Spanish speech understanding under realistic acoustic conditions has received particularly little attention. We introduce ESCUCHA, the first Spanish speech understanding benchmark designed to evaluate LALMs across heterogeneous acoustic conditions and reasoning abilities. ESCUCHA comprises 1,000 human-curated questions paired with audio, totaling 162.9 hours sourced directly ``from the wild'' rather than drawn from existing datasets, with durations ranging from a few seconds to over 80 minutes. The benchmark emphasizes reasoning, spanning 9 perceptual and 10 reasoning categories, and it captures linguistic diversity through multiple Spanish accents and non-normative speech. ESCUCHA further includes multi-audio questions, spoken questions, and audio instructions, and it flags which questions support open-ended evaluation. Benchmarking several state-of-the-art multimodal and speech models reveals substantial performance gaps relative to trained humans.

Related papers

• SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia [20.33019395334099]
  We introduce SEA-SpeechBench, the first large-scale benchmark that evaluates speech understanding in 11 Southeast Asian languages.<n>Our benchmark comprises 9 diverse tasks across 3 categories: speech processing (automatic speech recognition, speech translation, spoken question answering), paralinguistic analysis (emotion, gender, age, speaker recognition), and temporal understanding.<n>We implement multilingual prompting in both native SEA languages and English to reflect user interactions with audio-language models.
  arXiv  Detail & Related papers  (2026-09-09T03:39:18Z)
• S-DiverSe: Spanish Diverse Speech [67.47251829943015]
  S-DiverSe is a corpus of 3.2 hours of in-the-wild Spanish speech from 22 speakers with amyotrophic lateral sclerosis, Parkinson's disease, and stroke.<n>The dataset contains 444 manually transcribed audio segments with metadata on speaker sex, disease type, and intelligibility.
  arXiv  Detail & Related papers  (2026-07-03T11:23:43Z)
• MUSCAT: MUltilingual, SCientific ConversATion Benchmark [60.87925076316812]
  The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
  arXiv  Detail & Related papers  (2026-04-17T10:39:01Z)
• Do What I Say: A Spoken Prompt Dataset for Instruction-Following [64.72383858691795]
  DoWhatISay (DOWIS) is a dataset of human-recorded spoken and written prompts.<n>It provides 10 prompt variants per task-language pair, across five styles.<n>Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type.
  arXiv  Detail & Related papers  (2026-03-10T16:39:46Z)
• Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations [21.050704978484784]
  We construct and release the BiasInEar dataset, a speech-augmented benchmark based on Global MMLU Lite.<n>We evaluate nine representative models under linguistic (language and accent), demographic (gender), and structural (option order) perturbations.<n>Our findings reveal that MLLMs are relatively robust to demographic factors but highly sensitive to language and option order, suggesting that speech can amplify existing structural biases.
  arXiv  Detail & Related papers  (2026-02-01T05:34:34Z)
• Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models [38.608158064184366]
  We standardize and annotate the largest spoken Singlish corpus, introducing the Multitask National Speech Corpus (MNSC)<n>These datasets support diverse tasks, including Automatic Speech Recognition (ASR), Spoken Question Answering (SQA), Spoken Dialogue Summarization (SDS) and Paralinguistic Question Answering (PQA)<n>We propose SingAudioLLM, a multi-task multimodal model leveraging multimodal large language models to handle these tasks concurrently.
  arXiv  Detail & Related papers  (2025-01-02T03:28:52Z)
• Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models [58.43486430996411]
  Large Audio-Language Models (LALMs) have recently unlocked audio dialogue capabilities, enabling direct spoken exchanges with humans.<n>We propose an Audio Dialogue Understanding Benchmark (ADU-Bench) to evaluate the performance of LALMs in the open-ended audio dialogue understanding.<n>ADU-Bench includes over 20,000 open-ended audio dialogues for the assessment of LALMs.
  arXiv  Detail & Related papers  (2024-12-06T16:34:15Z)
• Do Audio-Language Models Understand Linguistic Variations? [42.17718387132912]
  Open-vocabulary audio language models (ALMs) represent a promising new paradigm for audio-text retrieval using natural language queries.<n>We propose RobustCLAP, a novel and compute-efficient technique to learn audio-language representations to linguistic variations.
  arXiv  Detail & Related papers  (2024-10-21T20:55:33Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)
• LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild [0.0]
  This paper presents a semi-automatically annotated audiovisual database to deal with unconstrained natural Spanish. Results for both speaker-dependent and speaker-independent scenarios are reported using Hidden Markov Models.
  arXiv  Detail & Related papers  (2023-11-21T09:12:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.