Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)

• URL: http://arxiv.org/abs/2606.07608v1
• Date: Fri, 29 May 2026 13:43:57 GMT
• Title: Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)
• Abstract summary: We fine-tune OpenAI's Whisper large-v3 for Swiss German ASR using 1,367 hours of broadcast speech paired with Standard German subtitles as weak supervision.<n>Our best model achieves 25.6% measured WER on the All Swiss German Dialects Test Set (ASGDTS) in an honest evaluation on strictly disjoint data.<n>A vanilla Whisper model self-trained on the ASGDTS test set with zero Swiss German data achieves 13.88% WER, surpassing all published systems.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a systematic study of fine-tuning OpenAI's Whisper large-v3 for Swiss German ASR, using 1,367 hours of broadcast speech paired with Standard German subtitles as weak supervision. Through 16 iterative training runs on an NVIDIA DGX Spark (Grace Blackwell, 128 GB unified memory, up to 1 PFLOP FP4), we compare LoRA and full fine-tuning of the 1.55B-parameter model, investigate hallucination root causes, and quantify the effect of data quality, subtitle alignment, and training strategy. Our best model achieves 25.6% measured WER on the All Swiss German Dialects Test Set (ASGDTS) in an honest evaluation on strictly disjoint data. A harmonized error analysis separating genuine errors from valid stylistic variation (tense, word order, Swiss orthography) yields a content WER (cWER) of 13.8%, counting only actual recognition failures. Bias-corrected estimation reduces this to 8.5%, suggesting the true error rate is roughly one third of measured WER. We demonstrate that published state-of-the-art Swiss German ASR results (17.1-17.5% WER) are inflated by benchmark contamination: a vanilla Whisper model self-trained on the ASGDTS test set with zero Swiss German data achieves 13.88% WER, surpassing all published systems. Experiments with Phi-4-multimodal show an even stronger memorization effect (3.9% WER), revealing that the benchmark primarily measures convention matching rather than dialectal comprehension. We release two models, a LoRA adapter (25.32% WER, 13.9% cWER) and a full fine-tuned model (25.60% WER, 13.8% cWER), among the few publicly available, honestly evaluated Whisper models for Swiss German, under Apache 2.0 with full reproducibility, requiring no institutional data agreements.

Related papers

• Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models [1.4095958360608893]
  This paper presents a controlled, fine-tuned Whisper-based Assamese ASR system trained on the Mozilla Common Voice 24.0-Assamese corpus.<n>A hardware-aware optimized training pipeline is implemented for resource-constrained environments.
  arXiv  Detail & Related papers  (2026-07-19T09:54:52Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Evaluating Commercial AI Chatbots as News Intermediaries [85.32040752972836]
  Best systems achieve over 90% multiple-choice accuracy on questions about events reported hours earlier.<n>Every model achieves its lowest accuracy on Hindi.<n> retrieval, not reasoning, failures drive over 70% of all errors.
  arXiv  Detail & Related papers  (2026-05-21T17:42:07Z)
• BlasBench: An Open Benchmark for Irish Speech Recognition [0.0]
  Existing benchmarks include Irish among dozens of languages but apply no Irish-aware text normalisation.<n>We introduce BlasBench, an open evaluation harness that provides a standalone Irish-aware normaliser preserving fadas, lenition, and eclipsis.<n>We pilot this by benchmarking 12 systems across four architecture families on Common Voice ga-IE and FLEURS ga-IE.
  arXiv  Detail & Related papers  (2026-04-12T17:17:54Z)
• Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation [0.0]
  Pashto is spoken by approximately 60--80 million people but has no published benchmarks for multilingual automatic speech recognition (ASR) on any shared public test set.<n>This paper reports the first reproducible multi-model evaluation on public Pashto data, covering zero-shot ASR, script-level failure, and cross-domain evaluation of fine-tuned models.
  arXiv  Detail & Related papers  (2026-04-06T11:23:42Z)
• Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks [0.0]
  No existing benchmark evaluates frontier model performance on applied Swiss regulatory tasks.<n>I introduce Swiss-Bench SBP, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains.<n>I evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework.
  arXiv  Detail & Related papers  (2026-03-24T18:41:07Z)
• Qwen3-ASR Technical Report [71.87071808763484]
  We introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model.<n>Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects.
  arXiv  Detail & Related papers  (2026-01-29T06:58:13Z)
• The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties [107.57160730151975]
  We construct a new test suite that consists of data from 200+ languages, accents, and dialects to evaluate SOTA multilingual speech models.<n>The best-performing submission achieved an absolute improvement in LID accuracy of 23% and a reduction in CER of 18%.<n>On accented and dialectal data, the best submission obtained 30.2% lower CER and 15.7% higher LID accuracy.
  arXiv  Detail & Related papers  (2025-09-08T18:42:36Z)
• Fact or Facsimile? Evaluating the Factual Robustness of Modern Retrievers [34.31192184496381]
  Dense retrievers and rerankers are central to retrieval-augmented generation (RAG) pipelines.<n>We evaluate how much factual competence these components inherit or lose from large language models (LLMs) they are based on.<n>For every embedding model, cosine-similarity scores between queries and correct completions are significantly higher than those for incorrect ones.
  arXiv  Detail & Related papers  (2025-08-28T04:13:51Z)
• Advancing STT for Low-Resource Real-World Speech [0.0]
  This paper introduces the new SRB-300 dataset, a 300-hour annotated speech corpus.<n>It captures spontaneous speech across all major Swiss dialects recorded in various realistic environments.<n>We fine-tuned multiple OpenAI Whisper models on the SRB-300 dataset, achieving notable enhancements over previous zero-shot performance metrics.
  arXiv  Detail & Related papers  (2025-06-10T14:22:48Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• Exploration of End-to-End ASR for OpenSTT -- Russian Open Speech-to-Text Dataset [73.66530509749305]
  This paper presents an exploration of end-to-end automatic speech recognition systems (ASR) for the largest open-source Russian language data set -- OpenSTT. We evaluate different existing end-to-end approaches such as joint CTC/Attention, RNN-Transducer, and Transformer. For the three available validation sets (phone calls, YouTube, and books), our best end-to-end model achieves word error rate (WER) of 34.8%, 19.1%, and 18.1%, respectively.
  arXiv  Detail & Related papers  (2020-06-15T10:35:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.