Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German

• URL: http://arxiv.org/abs/2605.19069v3
• Date: Fri, 22 May 2026 17:14:41 GMT
• Title: Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German
• Abstract summary: We present a benchmark evaluating five commercial ASR providers across four language pairs.<n>We evaluate on both WER and BERTScore, showing that while both metrics agree on the ordinal ranking of systems, WER inflates the magnitude of quality gaps.<n>ElevenLabs Scribe v2 achieves the lowest WER (13.2% overall) and leads on BERTScore (0.936 overall)
• Score: 0.34797121357690153
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code-switching -- the natural alternation between two languages within a single utterance -- remains one of the most challenging and under-studied conditions for automatic speech recognition (ASR). We present a benchmark evaluating five commercial ASR providers across four language pairs: Egyptian Arabic--English, Saudi Arabic (Najdi/Hijazi)--English, Persian (Farsi)--English, and German--English, comprising 300 samples per pair selected by a two-stage pipeline combining heuristic filtering with a GPT-4o and Gemini 1.5 Pro ensemble scorer, reducing LLM costs by $\approx$91\%. We evaluate on both WER and BERTScore, showing that while both metrics agree on the ordinal ranking of systems for all Arabic and Persian pairs ($τ= 1.0$), WER inflates the magnitude of quality gaps by approximately 3$\times$ by penalising semantically correct transliteration choices. ElevenLabs Scribe v2 achieves the lowest WER (13.2\% overall) and leads on BERTScore (0.936 overall). Difficulty-stratified analysis reveals performance gaps masked by aggregate averages, and BERT embedding projections confirm semantic proximity between reference and hypothesis despite surface-level script differences. The dataset is publicly available at https://huggingface.co/datasets/Perle-ai/ASR_Code_Switch.

Related papers

• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• MUSCAT: MUltilingual, SCientific ConversATion Benchmark [60.87925076316812]
  The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
  arXiv  Detail & Related papers  (2026-04-17T10:39:01Z)
• Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation [0.8253953000831505]
  We localize the Agent-as-a-Judge prompt stack to five typologically diverse languages (English, Arabic, Turkish, Chinese, Hindi)<n>We evaluate 55 DevAI development tasks across three developer-agent frameworks and six judge backbones, totaling 4950 judge runs.<n>GPT-4o achieves the highest satisfaction in English (44.72%), while Gemini leads in Arabic (51.72%, $p0.001$ vs. GPT-4o and Hindi)
  arXiv  Detail & Related papers  (2026-04-06T08:54:16Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages? [1.071318785217926]
  We present a benchmark spanning 12 Indian languages and four evaluation tasks.<n>E5-Large-Instruct achieves the highest Recall@1 of 27.4% on monolingual retrieval and 20.7% on cross-lingual transfer.<n>For classification, LaBSE attains 75.3% AUROC with strong calibration.
  arXiv  Detail & Related papers  (2026-01-15T09:10:14Z)
• AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3 [4.284434049360481]
  We present AraToken, an Arabic-optimized tokenizer built on SentencePiece Unigram algorithm.<n>We show that SentencePiece with normalization achieves 18% lower fertility (1.199 vs 1.35 tokens/word) compared to unnormalized baselines.<n>Our experiments show that LEP reduces evaluation loss from 8.28 to 2.43 within 800 training steps on 100K Arabic samples.
  arXiv  Detail & Related papers  (2025-12-20T15:32:10Z)
• ParsVoice: A Large-Scale Multi-Speaker Persian Speech Corpus for Text-to-Speech Synthesis [3.763275651955603]
  Existing Persian speech datasets are typically smaller than their English counterparts.<n>ParsVoice is the largest Persian speech corpus designed specifically for text-to-speech applications.
  arXiv  Detail & Related papers  (2025-10-12T19:33:11Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• Large Language Models for Persian $ \leftrightarrow $ English Idiom Translation [5.689194193929357]
  Large language models (LLMs) have shown superior capabilities in translating figurative language compared to neural machine translation (NMT) systems.<n>This paper introduces two parallel datasets of sentences containing idiomatic expressions for Persian$rightarrow$English and English$rightarrow$Persian translations.<n>We evaluate various open- and closed-source LLMs, NMT models, and their combinations.<n>Experiments reveal that Claude-3.5-Sonnet delivers outstanding results in both translation directions.
  arXiv  Detail & Related papers  (2024-12-13T09:29:27Z)
• Phonology-Guided Speech-to-Speech Translation for African Languages [2.7624021966289605]
  We present a prosody-guided framework for speech-to-speech translation (S2ST)<n>We show that emphwithin-phylum language pairs exhibit 30--40% lower pause and over 3$times$ higher onset/offset correlation compared to cross-phylum pairs.<n>We also release a three-tier, transcript-free BLEU suite (M1--M3) that correlates strongly with human judgments.
  arXiv  Detail & Related papers  (2024-10-30T09:44:52Z)
• Exploring SSL Discrete Tokens for Multilingual ASR [29.059205486691745]
  This study presents a comparison of discrete tokens generated by various leading SSL models across multiple language domains. Experimental results demonstrate that discrete tokens achieve comparable results against systems trained on Fbank features in ASR tasks.
  arXiv  Detail & Related papers  (2024-09-13T13:13:39Z)
• BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric [66.73705349465207]
  End-to-end speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. We propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems.
  arXiv  Detail & Related papers  (2022-12-16T14:00:26Z)
• Bilingual End-to-End ASR with Byte-Level Subwords [4.268218327369146]
  We study different representations including character-level, byte-level, byte pair encoding (BPE), and byte-level byte pair encoding (BBPE) We focus on developing a single end-to-end model to support utterance-based bilingual ASR, where speakers do not alternate between two languages in a single utterance but may change languages across utterances. We find that BBPE with penalty schemes can improve utterance-based bilingual ASR performance by 2% to 5% relative even with smaller number of outputs and fewer parameters.
  arXiv  Detail & Related papers  (2022-05-01T15:01:01Z)
• Arabic Speech Recognition by End-to-End, Modular Systems and Human [56.96327247226586]
  We perform a comprehensive benchmarking for end-to-end transformer ASR, modular HMM-DNN ASR, and human speech recognition. For ASR the end-to-end work led to 12.5%, 27.5%, 23.8% WER; a new performance milestone for the MGB2, MGB3, and MGB5 challenges respectively. Our results suggest that human performance in the Arabic language is still considerably better than the machine with an absolute WER gap of 3.6% on average.
  arXiv  Detail & Related papers  (2021-01-21T05:55:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.