MUSCAT: MUltilingual, SCientific ConversATion Benchmark

• URL: http://arxiv.org/abs/2604.15929v1
• Date: Fri, 17 Apr 2026 10:39:01 GMT
• Title: MUSCAT: MUltilingual, SCientific ConversATion Benchmark
• Abstract summary: The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
• Score: 60.87925076316812
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages, creating the experience as though everyone were a multilingual speaker. To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching. However, there is currently no dataset benchmarking this situation. We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges. The benchmark consists of bilingual discussions on scientific papers between multiple speakers, each conversing in a different language. We provide a standard evaluation framework, beyond Word Error Rate (WER) enabling consistent comparison of ASR performance across languages. Experimental results demonstrate that the proposed dataset is still an open challenge for state-of-the-art ASR systems. The dataset is available in https://huggingface.co/datasets/goodpiku/muscat-eval \\ \newline \Keywords{multilingual, speech recognition, audio segmentation, speaker diarization}

Related papers

• Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seeking [72.66384771245389]
  This work introduces HEALTHDIAL, a large-scale, multilingual, and multi-parallel dataset for developing and evaluating retrieval-augmented generation (RAG)-based spoken dialogue systems.<n>The dataset comprises 6,000 information-seeking dialogues (1,500 per language) grounded in trusted content from the World Health Organization (WHO) and 163 hours of user speech recorded from native speakers of diverse dialects.
  arXiv  Detail & Related papers  (2026-05-28T15:47:09Z)
• UniCoM: A Universal Code-Switching Speech Generator [19.893429976826464]
  Code-Switching (CS), the alternation between two or more languages within a single speaker's utterances, is common in real-world conversations.<n>We propose Universal Code-Mixer (UniCoM), a novel pipeline for generating high-quality, natural CS samples.<n>We construct Code-Switching FLEURS (CS-FLEURS), a multilingual CS corpus designed for automatic speech recognition (ASR) and speech-to-text translation (S2TT)
  arXiv  Detail & Related papers  (2025-08-21T05:11:21Z)
• SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset [34.40254709148148]
  Code-Switching (CS) is the alternating use of two or more languages within a conversation or utterance.<n>This linguistic phenomenon poses challenges for Automatic Speech Recognition (ASR) systems.<n>textbfSwitchLingua is the first large-scale multilingual and multi-ethnic code-switching dataset.
  arXiv  Detail & Related papers  (2025-05-30T05:54:46Z)
• Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models [38.608158064184366]
  We standardize and annotate the largest spoken Singlish corpus, introducing the Multitask National Speech Corpus (MNSC)<n>These datasets support diverse tasks, including Automatic Speech Recognition (ASR), Spoken Question Answering (SQA), Spoken Dialogue Summarization (SDS) and Paralinguistic Question Answering (PQA)<n>We propose SingAudioLLM, a multi-task multimodal model leveraging multimodal large language models to handle these tasks concurrently.
  arXiv  Detail & Related papers  (2025-01-02T03:28:52Z)
• CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval [0.9023847175654603]
  CLASP (Contrastive Language-Speech Pretraining) is a multilingual representation tailored for audio-text information retrieval.<n>During training, we utilize our newly introduced speech-text dataset, which encompasses 15 diverse categories ranging from fiction to religion.<n>Our evaluations across multiple languages demonstrate that CLASP establishes new benchmarks in HITS@1, MRR, and meanR metrics.
  arXiv  Detail & Related papers  (2024-12-17T16:38:10Z)
• Parrot: Multilingual Visual Instruction Tuning [66.65963606552839]
  Existing methods typically align vision encoders with Multimodal Large Language Models (MLLMs) via supervised fine-tuning (SFT)<n>We propose PARROT, a novel approach that leverages textual guidance for visual token alignment at the language level.<n>We introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions.
  arXiv  Detail & Related papers  (2024-06-04T17:56:28Z)
• ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech [58.93395189153713]
  We extend the pretraining method for cross-lingual multi-speaker speech synthesis tasks. We propose a speech-text joint pretraining framework, where we randomly mask the spectrogram and the phonemes. Our model shows great improvements over speaker-embedding-based multi-speaker TTS methods.
  arXiv  Detail & Related papers  (2022-11-07T13:35:16Z)
• Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition [62.997667081978825]
  Code-Switching (CS) is referred to the phenomenon of alternately using words and phrases from different languages. We propose a simple yet effective data augmentation in which audio and corresponding labels of different source languages are transcribed. We show that this augmentation can even improve the model's performance on inter-sentential language switches not seen during training by 5,03% WER.
  arXiv  Detail & Related papers  (2022-10-17T12:15:57Z)
• The Conversational Short-phrase Speaker Diarization (CSSD) Task: Dataset, Evaluation Metric and Baselines [63.86406909879314]
  This paper describes the Conversational Short-phrases Speaker Diarization (CSSD) task. It consists of training and testing datasets, evaluation metric and baselines. In the metric aspect, we design the new conversational DER (CDER) evaluation metric, which calculates the SD accuracy at the utterance level.
  arXiv  Detail & Related papers  (2022-08-17T03:26:23Z)
• Cross-lingual Multispeaker Text-to-Speech under Limited-Data Scenario [10.779568857641928]
  This paper presents an extension on Tacotron2 to achieve bilingual multispeaker speech synthesis. We achieve cross-lingual synthesis, including code-switching cases, between English and Mandarin for monolingual speakers.
  arXiv  Detail & Related papers  (2020-05-21T03:03:34Z)
• That Sounds Familiar: an Analysis of Phonetic Representations Transfer Across Languages [72.9927937955371]
  We use the resources existing in other languages to train a multilingual automatic speech recognition model. We observe significant improvements across all languages in the multilingual setting, and stark degradation in the crosslingual setting. Our analysis uncovered that even the phones that are unique to a single language can benefit greatly from adding training data from other languages.
  arXiv  Detail & Related papers  (2020-05-16T22:28:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.