Qualitative Evaluation of Language Model Rescoring in Automatic Speech Recognition

• URL: http://arxiv.org/abs/2604.27533v1
• Date: Thu, 30 Apr 2026 07:37:16 GMT
• Title: Qualitative Evaluation of Language Model Rescoring in Automatic Speech Recognition
• Abstract summary: We study the impact of rescoring using language models in ASR systems by means of several metrics.<n>These metrics illustrate the linguistic contributions of the language models that are applied during a posterior rescoring step on transcription hypotheses.
• Score: 6.732553711612198
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating automatic speech recognition (ASR) systems is a classical but difficult and still open problem, which often boils down to focusing only on the word error rate (WER). However, this metric suffers from many limitations and does not allow an in-depth analysis of automatic transcription errors. In this paper, we propose to study and understand the impact of rescoring using language models in ASR systems by means of several metrics often used in other natural language processing (NLP) tasks in addition to the WER. In particular, we introduce two measures related to morpho-syntactic and semantic aspects of transcribed words: 1) the POSER (Part-of-speech Error Rate), which should highlight the grammatical aspects, and 2) the EmbER (Embedding Error Rate), a measurement that modifies the WER by providing a weighting according to the semantic distance of the wrongly transcribed words. These metrics illustrate the linguistic contributions of the language models that are applied during a posterior rescoring step on transcription hypotheses.

Related papers

• A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition [12.151730134044696]
  We propose a paradigm that consists in incorporating a chosen metric into it in order to obtain an equivalent of the error rate: a Minimum Edit Distance (minED)<n>This approach parallels transcription errors with their human perception, also allowing an original study of the severity of these errors from a human perspective.
  arXiv  Detail & Related papers  (2026-05-05T12:09:12Z)
• HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics [11.326630899294004]
  We present an original French manually annotated data set in terms of human perception of transcription errors produced by various ASR systems.<n>We investigated the relationship between human preferences and various ASR evaluation metrics, including lexical and embedding-based ones.
  arXiv  Detail & Related papers  (2026-04-30T07:48:20Z)
• WESR: Scaling and Evaluating Word-level Event-Speech Recognition [59.21814194620928]
  Speech conveys not only linguistic information but also rich non-verbal vocal events such as laughing and crying.<n>We develop a refined taxonomy of 21 vocal events, with a new categorization into discrete (standalone) versus continuous (mixed with speech) types.<n>Based on the refined taxonomy, we introduce WESR-Bench, an expert-annotated evaluation set (900+ utterances) with a novel position-aware protocol.
  arXiv  Detail & Related papers  (2026-01-08T02:23:21Z)
• Context Biasing for Pronunciations-Orthography Mismatch in Automatic Speech Recognition [61.601626186678146]
  We propose a method which allows corrections of substitution errors to improve the recognition accuracy of challenging words.<n>We show that with this method we get a relative improvement in biased word error rate of up to 8%, while maintaining a competitive overall word error rate.
  arXiv  Detail & Related papers  (2025-06-23T14:42:03Z)
• A Benchmark of French ASR Systems Based on Error Severity [6.657432034629865]
  A new evaluation is proposed that categorizes errors into four levels of severity.<n>This metric is applied to a benchmark of 10 state-of-the-art ASR systems on French language.
  arXiv  Detail & Related papers  (2025-01-18T21:07:18Z)
• Speaker Tagging Correction With Non-Autoregressive Language Models [0.0]
  We propose a speaker tagging correction system based on a non-autoregressive language model. We show that the employed error correction approach leads to reductions in word diarization error rate (WDER) on two datasets.
  arXiv  Detail & Related papers  (2024-08-30T11:02:17Z)
• Towards Unsupervised Speech Recognition Without Pronunciation Models [57.222729245842054]
  In this article, we tackle the challenge of developing ASR systems without paired speech and text corpora.<n>We experimentally demonstrate that an unsupervised speech recognizer can emerge from joint speech-to-speech and text-to-text masked token-infilling.<n>This innovative model surpasses the performance of previous unsupervised ASR models under the lexicon-free setting.
  arXiv  Detail & Related papers  (2024-06-12T16:30:58Z)
• Continuously Learning New Words in Automatic Speech Recognition [56.972851337263755]
  We propose a self-supervised continual learning approach for Automatic Speech Recognition.<n>We use a memory-enhanced ASR model from the literature to decode new words from the slides.<n>We show that with this approach, we obtain increasing performance on the new words when they occur more frequently.
  arXiv  Detail & Related papers  (2024-01-09T10:39:17Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Unsupervised Language agnostic WER Standardization [4.768240090076601]
  We propose an automatic WER normalization system consisting of two modules: spelling normalization and segmentation normalization. Experiments with ASR on 35K utterances across four languages yielded an average WER reduction of 13.28%.
  arXiv  Detail & Related papers  (2023-03-09T05:50:54Z)
• Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale [52.663117551150954]
  A few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models.
  arXiv  Detail & Related papers  (2020-10-26T13:57:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.