SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

• URL: http://arxiv.org/abs/2609.09672v1
• Date: Wed, 09 Sep 2026 03:39:18 GMT
• Title: SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia
• Abstract summary: We introduce SEA-SpeechBench, the first large-scale benchmark that evaluates speech understanding in 11 Southeast Asian languages.<n>Our benchmark comprises 9 diverse tasks across 3 categories: speech processing (automatic speech recognition, speech translation, spoken question answering), paralinguistic analysis (emotion, gender, age, speaker recognition), and temporal understanding.<n>We implement multilingual prompting in both native SEA languages and English to reflect user interactions with audio-language models.
• Score: 20.33019395334099
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid advancement of audio and multimodal large language models has unlocked transformative speech understanding capabilities, yet evaluation frameworks remain predominantly English-centric, leaving Southeast Asian (SEA) languages critically underrepresented. We introduce SEA-SpeechBench, to the best of our knowledge, the first large-scale multitask benchmark that evaluates speech understanding in 11 SEA languages through 97,194 samples across 99 evaluation sets and 597 hours of curated audio data. Our benchmark comprises 9 diverse tasks across 3 categories: speech processing (automatic speech recognition, speech translation, spoken question answering), paralinguistic analysis (emotion, gender, age, speaker recognition), and temporal understanding, a novel dimension featuring timestamped content queries and temporal localization within extended audio sequences up to 3 minutes. We implement multilingual prompting in both native SEA languages and English to reflect user interactions with audio-language models. Evaluation of leading open-source and proprietary systems reveals marked performance gaps. Across all models, performance remains underwhelming on temporal understanding, emotion recognition, and speech translation. Prompting in low-resource languages such as Burmese and Tamil lags behind English by up to 41 percentage points. Our findings expose critical model limitations and underscore the need for inclusive model development. The SEA-SpeechBench benchmark is available at https://zwenyu.github.io/SEA-SpeechBench/.

Related papers

• ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions [66.93528077633728]
  ESCUCHA comprises 1,000 human-curated questions paired with audio, totaling 162.9 hours sourced directly from the wild'' rather than drawn from existing datasets.<n>The benchmark emphasizes reasoning, spanning 9 perceptual and 10 reasoning categories, and it captures linguistic diversity through multiple Spanish accents and non-normative speech.<n> Benchmarking several state-of-the-art multimodal and speech models reveals substantial performance gaps relative to trained humans.
  arXiv  Detail & Related papers  (2026-07-20T10:59:15Z)
• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects [29.32197370490759]
  PolySpeech-100 is a massive-scale benchmark designed to assess native-level' speech comprehension across 110 linguistic variants.<n>We employ a novel hybrid construction pipeline that augments gold-standard human recordings with instruction-driven synthetic speech.
  arXiv  Detail & Related papers  (2026-05-31T05:13:32Z)
• KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs [7.195547595036643]
  We propose two human-agent benchmark-construction frameworks for SpeechLM evaluation.<n>One transfers source-language SpokenQA benchmarks into target-language SpokenQA benchmarks, and the other converts target-language ASR corpora into audio understanding benchmarks.<n>Using these frameworks, we construct and publicly release three Korean speech benchmarks: KVoiceBench and KOpenAudioBench for Korean SpokenQA, and KMMAU for Korean audio understanding, comprising 12,345 samples in total.
  arXiv  Detail & Related papers  (2026-05-27T05:19:38Z)
• MUSCAT: MUltilingual, SCientific ConversATion Benchmark [60.87925076316812]
  The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
  arXiv  Detail & Related papers  (2026-04-17T10:39:01Z)
• SeaLLMs-Audio: Large Audio-Language Models for Southeast Asia [40.53123362174684]
  We introduce SeaLLMs-Audio, the first large audio-language model (LALM) tailored for multiple Southeast Asian languages.<n>SeaLLMs-Audio exhibits strong performance across diverse audio-centric tasks, spanning fine-grained audio understanding and voice-based interaction.<n>It supports a wide range of tasks, including audio analysis tasks such as Audio Captioning, Automatic Speech Recognition, Speech-to-Text Translation, Speech Emotion Recognition, Speech Question Answering, and Speech Summarization.
  arXiv  Detail & Related papers  (2025-11-03T15:32:58Z)
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark [42.58439306999647]
  MMSU comprises 5,000 meticulously curated audio-question-answer triplets across 47 distinct tasks.<n>We ground our benchmark in linguistic theory, including phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics.<n>MMSU establishes a new standard for comprehensive assessment of spoken language understanding.
  arXiv  Detail & Related papers  (2025-06-05T09:09:36Z)
• Long-Form Speech Generation with Spoken Language Models [64.29591880693468]
  Textless spoken language models struggle to generate plausible speech past tens of seconds.<n>We derive SpeechSSM, the first speech language model family to learn from and sample long-form spoken audio.<n>SpeechSSMs leverage recent advances in linear-time sequence modeling to greatly surpass current Transformer spoken LMs in coherence and efficiency.
  arXiv  Detail & Related papers  (2024-12-24T18:56:46Z)
• Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models [58.43486430996411]
  Large Audio-Language Models (LALMs) have recently unlocked audio dialogue capabilities, enabling direct spoken exchanges with humans.<n>We propose an Audio Dialogue Understanding Benchmark (ADU-Bench) to evaluate the performance of LALMs in the open-ended audio dialogue understanding.<n>ADU-Bench includes over 20,000 open-ended audio dialogues for the assessment of LALMs.
  arXiv  Detail & Related papers  (2024-12-06T16:34:15Z)
• SeamlessM4T: Massively Multilingual & Multimodal Machine Translation [90.71078166159295]
  We introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-text translation, and automatic speech recognition for up to 100 languages. We developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation.
  arXiv  Detail & Related papers  (2023-08-22T17:44:18Z)
• AudioPaLM: A Large Language Model That Can Speak and Listen [79.44757696533709]
  We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models. It can process and generate text and speech with applications including speech recognition and speech-to-speech translation.
  arXiv  Detail & Related papers  (2023-06-22T14:37:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.