"Sorry, I Didn't Catch That": How Speech Models Miss What Matters Most

• URL: http://arxiv.org/abs/2602.12249v2
• Date: Mon, 16 Feb 2026 22:39:51 GMT
• Title: "Sorry, I Didn't Catch That": How Speech Models Miss What Matters Most
• Authors: Kaitlyn Zhou, Martijn Bartelds, Federico Bianchi, James Zou,
• Abstract summary: We study the failure mode in a high-stakes task: the transcription of U.S. street names as spoken by U.S. participants.<n>We evaluate 15 models from OpenAI, Deepgram, Google, and Microsoft on recordings from linguistically diverse U.S. speakers and find an average transcription error rate of 44%.<n>To mitigate this harm, we introduce a synthetic data generation approach that produces diverse pronunciations of named entities using open-source text-to-speech models.
• Score: 30.735876729204012
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite speech recognition systems achieving low word error rates on standard benchmarks, they often fail on short, high-stakes utterances in real-world deployments. Here, we study this failure mode in a high-stakes task: the transcription of U.S. street names as spoken by U.S. participants. We evaluate 15 models from OpenAI, Deepgram, Google, and Microsoft on recordings from linguistically diverse U.S. speakers and find an average transcription error rate of 44%. We quantify the downstream impact of failed transcriptions by geographic locations and show that mis-transcriptions systematically cause errors for all speakers, but that routing distance errors are twice as large for non-English primary speakers compared to English primary speakers. To mitigate this harm, we introduce a synthetic data generation approach that produces diverse pronunciations of named entities using open-source text-to-speech models. Fine-tuning with less than 1,000 synthetic samples improves street name transcription accuracy by nearly 60% (relative to base models) for non-English primary speakers. Our results highlight a critical gap between benchmark performance and real-world reliability in speech systems and demonstrate a simple, scalable path to reducing high-stakes transcription errors.

Related papers

• Context Biasing for Pronunciations-Orthography Mismatch in Automatic Speech Recognition [61.601626186678146]
  We propose a method which allows corrections of substitution errors to improve the recognition accuracy of challenging words.<n>We show that with this method we get a relative improvement in biased word error rate of up to 8%, while maintaining a competitive overall word error rate.
  arXiv  Detail & Related papers  (2025-06-23T14:42:03Z)
• Speaker Tagging Correction With Non-Autoregressive Language Models [0.0]
  We propose a speaker tagging correction system based on a non-autoregressive language model. We show that the employed error correction approach leads to reductions in word diarization error rate (WDER) on two datasets.
  arXiv  Detail & Related papers  (2024-08-30T11:02:17Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correction [4.409889336732851]
  Speaker diarization (SD) is typically used with an automatic speech recognition (ASR) system to ascribe speaker labels to recognized words. This approach can lead to speaker errors especially around speaker turns and regions of speaker overlap. We propose a novel second-pass speaker error correction system using lexical information.
  arXiv  Detail & Related papers  (2023-06-15T17:47:41Z)
• Adversarial Training For Low-Resource Disfluency Correction [50.51901599433536]
  We propose an adversarially-trained sequence-tagging model for Disfluency Correction (DC) We show the benefit of our proposed technique, which crucially depends on synthetically generated disfluent data, by evaluating it for DC in three Indian languages. Our technique also performs well in removing stuttering disfluencies in ASR transcripts introduced by speech impairments.
  arXiv  Detail & Related papers  (2023-06-10T08:58:53Z)
• Simple and Effective Unsupervised Speech Translation [68.25022245914363]
  We study a simple and effective approach to build speech translation systems without labeled data. We present an unsupervised domain adaptation technique for pre-trained speech models. Experiments show that unsupervised speech-to-text translation outperforms the previous unsupervised state of the art.
  arXiv  Detail & Related papers  (2022-10-18T22:26:13Z)
• ASR Error Detection via Audio-Transcript entailment [1.3750624267664155]
  We propose an end-to-end approach for ASR error detection using audio-transcript entailment. The proposed model utilizes an acoustic encoder and a linguistic encoder to model the speech and transcript respectively. Our proposed model achieves classification error rates (CER) of 26.2% on all transcription errors and 23% on medical errors specifically, leading to improvements upon a strong baseline by 12% and 15.4%, respectively.
  arXiv  Detail & Related papers  (2022-07-22T02:47:15Z)
• Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need [18.446969150062586]
  Existing CAPT methods are not able to detect pronunciation errors with high accuracy. We present three innovative techniques based on phoneme-to-phoneme (P2P), text-to-speech (T2S), and speech-to-speech (S2S) conversion. We show that these techniques not only improve the accuracy of three machine learning models for detecting pronunciation errors but also help establish a new state-of-the-art in the field.
  arXiv  Detail & Related papers  (2022-07-02T08:33:33Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Unsupervised Speech Recognition [55.864459085947345]
  wav2vec-U, short for wav2vec Unsupervised, is a method to train speech recognition models without any labeled data. We leverage self-supervised speech representations to segment unlabeled audio and learn a mapping from these representations to phonemes via adversarial training. On the larger English Librispeech benchmark, wav2vec-U achieves a word error rate of 5.9 on test-other, rivaling some of the best published systems trained on 960 hours of labeled data from only two years ago.
  arXiv  Detail & Related papers  (2021-05-24T04:10:47Z)
• English Accent Accuracy Analysis in a State-of-the-Art Automatic Speech Recognition System [3.4888132404740797]
  We evaluate a state-of-the-art automatic speech recognition model, using unseen data from a corpus with a wide variety of labeled English accents. We show that there is indeed an accuracy bias in terms of accentual variety, favoring the accents most prevalent in the training corpus.
  arXiv  Detail & Related papers  (2021-05-09T08:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.