A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition

• URL: http://arxiv.org/abs/2605.03671v1
• Date: Tue, 05 May 2026 12:09:12 GMT
• Title: A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition
• Abstract summary: We propose a paradigm that consists in incorporating a chosen metric into it in order to obtain an equivalent of the error rate: a Minimum Edit Distance (minED)<n>This approach parallels transcription errors with their human perception, also allowing an original study of the severity of these errors from a human perspective.
• Score: 12.151730134044696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The most commonly used metrics for evaluating automatic speech transcriptions, namely Word Error Rate (WER) and Character Error Rate (CER), have been heavily criticized for their poor correlation to human perception and their inability to take into account linguistic and semantic information. While metric-based embeddings, seeking to approximate human perception, have been proposed, their scores remain difficult to interpret, unlike WER and CER. In this article, we overcome this problem by proposing a paradigm that consists in incorporating a chosen metric into it in order to obtain an equivalent of the error rate: a Minimum Edit Distance (minED). This approach parallels transcription errors with their human perception, also allowing an original study of the severity of these errors from a human perspective.

Related papers

• Qualitative Evaluation of Language Model Rescoring in Automatic Speech Recognition [6.732553711612198]
  We study the impact of rescoring using language models in ASR systems by means of several metrics.<n>These metrics illustrate the linguistic contributions of the language models that are applied during a posterior rescoring step on transcription hypotheses.
  arXiv  Detail & Related papers  (2026-04-30T07:37:16Z)
• Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement [19.427711407628024]
  Word-level quality estimation (WQE) aims to automatically identify fine-grained error spans in machine-translated outputs.<n>Modern WQE techniques are often expensive, involving prompting of large language models or ad-hoc training on large amounts of human-labeled data.
  arXiv  Detail & Related papers  (2025-05-29T07:20:36Z)
• A Benchmark of French ASR Systems Based on Error Severity [6.657432034629865]
  A new evaluation is proposed that categorizes errors into four levels of severity.<n>This metric is applied to a benchmark of 10 state-of-the-art ASR systems on French language.
  arXiv  Detail & Related papers  (2025-01-18T21:07:18Z)
• What's under the hood: Investigating Automatic Metrics on Meeting Summarization [7.234196390284036]
  Meeting summarization has become a critical task considering the increase in online interactions. Current default-used metrics struggle to capture observable errors, showing weak to mid-correlations. Only a subset reacts accurately to specific errors, while most correlations show either unresponsiveness or failure to reflect the error's impact on summary quality.
  arXiv  Detail & Related papers  (2024-04-17T07:15:07Z)
• Cobra Effect in Reference-Free Image Captioning Metrics [58.438648377314436]
  A proliferation of reference-free methods, leveraging visual-language pre-trained models (VLMs), has emerged. In this paper, we study if there are any deficiencies in reference-free metrics. We employ GPT-4V as an evaluative tool to assess generated sentences and the result reveals that our approach achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2024-02-18T12:36:23Z)
• Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets [92.38654521870444]
  We introduce ACES, a contrastive challenge set spanning 146 language pairs. This dataset aims to discover whether metrics can identify 68 translation accuracy errors. We conduct a large-scale study by benchmarking ACES on 50 metrics submitted to the WMT 2022 and 2023 metrics shared tasks.
  arXiv  Detail & Related papers  (2024-01-29T17:17:42Z)
• BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training [64.37683359609308]
  In this study, we analyze various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. We find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm.
  arXiv  Detail & Related papers  (2023-07-06T16:59:30Z)
• MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types [68.76742370525234]
  We propose a new evaluation scheme to model human judgments in 7 NLP tasks, based on the fine-grained mismatches between a pair of texts. Inspired by the recent efforts in several NLP tasks for fine-grained evaluation, we introduce a set of 13 mismatch error types. We show that the mismatch errors between the sentence pairs on the held-out datasets from 7 NLP tasks align well with the human evaluation.
  arXiv  Detail & Related papers  (2023-06-18T01:38:53Z)
• BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation [12.407789866525079]
  We show that by using additional information during training, such as sentence-level features and word-level tags, the trained metrics improve their capability to penalize translations with specific troublesome phenomena. We show that by using additional information during training, such as sentence-level features and word-level tags, the trained metrics improve their capability to penalize translations with specific troublesome phenomena.
  arXiv  Detail & Related papers  (2023-05-30T15:50:46Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• On the Blind Spots of Model-Based Evaluation Metrics for Text Generation [79.01422521024834]
  We explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics. We design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores. Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics.
  arXiv  Detail & Related papers  (2022-12-20T06:24:25Z)
• Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics [64.88815792555451]
  We show that current methods for judging metrics are highly sensitive to the translations used for assessment. We develop a method for thresholding performance improvement under an automatic metric against human judgements.
  arXiv  Detail & Related papers  (2020-06-11T09:12:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.