KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness

• URL: http://arxiv.org/abs/2604.19782v1
• Date: Mon, 30 Mar 2026 06:13:27 GMT
• Title: KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness
• Abstract summary: KoALa-Bench is a benchmark for evaluating Korean speech understanding and speech faithfulness of large audio language models (LALMs)<n>Four tasks evaluate fundamental speech understanding capabilities, including automatic speech recognition, speech translation, speech question answering, and speech instruction following.<n>The remaining two tasks evaluate speech faithfulness, motivated by our observation that several LALMs often fail to fully leverage the speech modality.
• Score: 11.620492203109983
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in large audio language models (LALMs) have enabled multilingual speech understanding. However, benchmarks for evaluating LALMs remain scarce for non-English languages, with Korean being one such underexplored case. In this paper, we introduce KoALa-Bench, a comprehensive benchmark for evaluating Korean speech understanding and speech faithfulness of LALMs. In particular, KoALa-Bench comprises six tasks. Four tasks evaluate fundamental speech understanding capabilities, including automatic speech recognition, speech translation, speech question answering, and speech instruction following, while the remaining two tasks evaluate speech faithfulness, motivated by our observation that several LALMs often fail to fully leverage the speech modality. Furthermore, to reflect Korea-specific knowledge, our benchmark incorporates listening questions from the Korean college scholastic ability test as well as content covering Korean cultural domains. We conduct extensive experiments across six models, including both white-box and black-box ones. Our benchmark, evaluation code, and leaderboard are publicly available at https://ksbench.github.io/Korean-Benchmark/.

Related papers

• KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs [7.195547595036643]
  We propose two human-agent benchmark-construction frameworks for SpeechLM evaluation.<n>One transfers source-language SpokenQA benchmarks into target-language SpokenQA benchmarks, and the other converts target-language ASR corpora into audio understanding benchmarks.<n>Using these frameworks, we construct and publicly release three Korean speech benchmarks: KVoiceBench and KOpenAudioBench for Korean SpokenQA, and KMMAU for Korean audio understanding, comprising 12,345 samples in total.
  arXiv  Detail & Related papers  (2026-05-27T05:19:38Z)
• KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models [36.90941464587649]
  We introduce the Korean Instruction-following Task Evaluation (KITE), a benchmark designed to evaluate both general and Korean-specific instructions.<n>Unlike existing Korean benchmarks that focus mainly on factual knowledge or multiple-choice testing, KITE directly targets diverse, open-ended instruction-following tasks.
  arXiv  Detail & Related papers  (2025-10-17T11:45:15Z)
• VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents [53.33704332801441]
  Large audio language models (LALMs) have greatly enhanced multimodal conversational systems.<n>Existing benchmarks are mainly English-centric, rely on synthetic speech, and lack comprehensive, discriminative evaluation.<n>We present Voice Chat Bot Bench (VCB Bench) -- a high-quality Chinese benchmark built entirely on real human speech.
  arXiv  Detail & Related papers  (2025-10-13T07:45:52Z)
• Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs [59.230858581944425]
  Two dominant approaches have emerged for speech processing: discrete tokens and continuous features.<n>We compare self-supervised learning (SSL)-based discrete and continuous features under the same experimental settings.<n>Our findings reveal that continuous features generally outperform discrete tokens in various tasks.
  arXiv  Detail & Related papers  (2025-08-25T10:16:07Z)
• Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean [8.072947878765941]
  KoGEM is designed to assess the linguistic competence of LLMs and humans in Korean.<n>It consists of 1.5k multiple-choice QA pairs covering five main categories and 16 subcategories.
  arXiv  Detail & Related papers  (2025-06-02T01:27:46Z)
• Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models [49.1574468325115]
  We introduce Speech-IFeval, an evaluation framework designed to assess instruction-following capabilities.<n>Recent SLMs integrate speech perception with large language models (LLMs), often degrading textual capabilities due to speech-centric training.<n>Our findings show that most SLMs struggle with even basic instructions, performing far worse than text-based LLMs.
  arXiv  Detail & Related papers  (2025-05-25T08:37:55Z)
• EXECUTE: A Multilingual Benchmark for LLM Token Understanding [54.70665106141121]
  Tests across multiple languages reveal that challenges in other languages are not always on the character level as in English.<n>We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs' understanding of character components.
  arXiv  Detail & Related papers  (2025-05-23T11:56:48Z)
• Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models [58.43486430996411]
  Large Audio-Language Models (LALMs) have recently unlocked audio dialogue capabilities, enabling direct spoken exchanges with humans.<n>We propose an Audio Dialogue Understanding Benchmark (ADU-Bench) to evaluate the performance of LALMs in the open-ended audio dialogue understanding.<n>ADU-Bench includes over 20,000 open-ended audio dialogues for the assessment of LALMs.
  arXiv  Detail & Related papers  (2024-12-06T16:34:15Z)
• MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models [65.10456412127405]
  MLaKE is a benchmark for the adaptability of knowledge editing methods across five languages.<n>MLaKE aggregates fact chains from Wikipedia across languages and generates questions in both free-form and multiple-choice.<n>We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE.
  arXiv  Detail & Related papers  (2024-04-07T15:23:28Z)
• Pragmatic Competence Evaluation of Large Language Models for the Korean Language [0.6757476692230009]
  This study evaluates how well Large Language Models (LLMs) understand context-dependent expressions from a pragmatic standpoint, specifically in Korean. We use both Multiple-Choice Questions (MCQs) for automatic evaluation and Open-Ended Questions (OEQs) assessed by human experts.
  arXiv  Detail & Related papers  (2024-03-19T12:21:20Z)
• HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models [0.0]
  We introduce the HAE-RAE Bench, a dataset curated to challenge models lacking Korean cultural and contextual depth. The dataset encompasses six downstream tasks across four domains: vocabulary, history, general knowledge, and reading comprehension.
  arXiv  Detail & Related papers  (2023-09-06T04:38:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.