VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context

• URL: http://arxiv.org/abs/2511.08230v2
• Date: Mon, 17 Nov 2025 04:39:12 GMT
• Title: VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context
• Abstract summary: We propose VocalBench-zh, an ability-level divided evaluation suite adapted to Mandarin context.<n>The evaluation experiment on 14 mainstream models reveals the common challenges for current routes.
• Score: 32.35011898483814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The development of multi-modal large language models (LLMs) leads to intelligent approaches capable of speech interactions. As one of the most widely spoken languages globally, Mandarin is supported by most models to enhance their applicability and reach. However, the scarcity of comprehensive speech-to-speech (S2S) benchmarks in Mandarin contexts impedes systematic evaluation for developers and hinders fair model comparison for users. In this work, we propose VocalBench-zh, an ability-level divided evaluation suite adapted to Mandarin context consisting of 10 well-crafted subsets and over 10K high-quality instances, covering 12 user-oriented characters. The evaluation experiment on 14 mainstream models reveals the common challenges for current routes, and highlights the need for new insights into next-generation speech interactive systems. The evaluation codes and datasets will be available at https://github.com/SJTU-OmniAgent/VocalBench-zh.

Related papers

• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects [29.32197370490759]
  PolySpeech-100 is a massive-scale benchmark designed to assess native-level' speech comprehension across 110 linguistic variants.<n>We employ a novel hybrid construction pipeline that augments gold-standard human recordings with instruction-driven synthetic speech.
  arXiv  Detail & Related papers  (2026-05-31T05:13:32Z)
• Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages [14.465157274862952]
  We present a controlled multidimensional pairwise evaluation framework for multilingual TTS.<n>We evaluate 7 state-of-the-art TTS systems and collect over 120K pairwise comparisons from over 1900 native raters.
  arXiv  Detail & Related papers  (2026-04-23T09:44:03Z)
• MUSCAT: MUltilingual, SCientific ConversATion Benchmark [60.87925076316812]
  The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
  arXiv  Detail & Related papers  (2026-04-17T10:39:01Z)
• Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning [5.770962296305264]
  We introduce a context-aware multilingual ASR framework that supports diverse languages and accents.<n>Our approach combines a frozen speech encoder and a decoder-only language model via a lightweight projection module.<n>We show that contextual input consistently improves recognition quality.
  arXiv  Detail & Related papers  (2026-03-06T17:37:06Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents [53.33704332801441]
  Large audio language models (LALMs) have greatly enhanced multimodal conversational systems.<n>Existing benchmarks are mainly English-centric, rely on synthetic speech, and lack comprehensive, discriminative evaluation.<n>We present Voice Chat Bot Bench (VCB Bench) -- a high-quality Chinese benchmark built entirely on real human speech.
  arXiv  Detail & Related papers  (2025-10-13T07:45:52Z)
• MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions [70.93364531054273]
  We introduce MultiVox, the first benchmark to evaluate the ability of voice assistants to integrate spoken and visual cues.<n>Specifically, MultiVox includes 1000 human-annotated and recorded speech dialogues that encompass diverse paralinguistic features.<n>Our evaluation on 10 state-of-the-art models reveals that, although humans excel at these tasks, current models consistently struggle to produce contextually grounded responses.
  arXiv  Detail & Related papers  (2025-07-14T23:20:42Z)
• Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR [23.285609467633865]
  This paper introduces the integration of language-specific bi-directional context into a speech large language model (SLLM) to improve multilingual continuous conversational automatic speech recognition (ASR)<n>We propose a character-level contextual masking strategy during training, which randomly removes portions of the context to enhance robustness and better emulate the flawed transcriptions that may occur during inference.
  arXiv  Detail & Related papers  (2025-06-16T12:03:23Z)
• In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties [24.74769794165231]
  We introduce a scalable framework that allows for in-context learning (ICL) in Phi-4 Multimodal.<n>We find that as few as 12 example utterances at inference time reduce word error rates by a relative 19.7%.<n>Overall, we find that our novel ICL adaptation scheme reveals a similar performance profile to human listeners.
  arXiv  Detail & Related papers  (2025-05-20T20:20:37Z)
• Classification of Spontaneous and Scripted Speech for Multilingual Audio [9.925703861731506]
  Distinguishing scripted from spontaneous speech is an essential tool for better understanding how speech styles influence speech processing research.<n>This paper addresses the challenge of building a classifier that generalises well across different formats and languages.<n>We systematically evaluate models ranging from traditional, handcrafted acoustic and prosodic features to advanced audio transformers.
  arXiv  Detail & Related papers  (2024-12-16T15:45:10Z)
• Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching [65.74653592668743]
  Finetuning self-supervised multilingual representations reduces absolute word error rates by up to 20%. In circumstances with limited training data finetuning self-supervised representations is a better performing and viable solution.
  arXiv  Detail & Related papers  (2023-11-25T17:05:21Z)
• SeamlessM4T: Massively Multilingual & Multimodal Machine Translation [90.71078166159295]
  We introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-text translation, and automatic speech recognition for up to 100 languages. We developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation.
  arXiv  Detail & Related papers  (2023-08-22T17:44:18Z)
• Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification [73.5497360800395]
  We develop an end-to-end system that supports multiple languages. We exploit knowledge from a pre-trained multi-lingual natural language processing model.
  arXiv  Detail & Related papers  (2021-09-28T04:43:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.