What do the metrics mean? A critical analysis of the use of Automated Evaluation Metrics in Interpreting

• URL: http://arxiv.org/abs/2601.05864v1
• Date: Fri, 09 Jan 2026 15:39:28 GMT
• Title: What do the metrics mean? A critical analysis of the use of Automated Evaluation Metrics in Interpreting
• Abstract summary: There is now a high demand for ways to quickly and efficiently measure the quality of any interpreting delivered.<n>This article examines these recently-proposed quality measurement methods and will discuss their suitability for measuring the quality of authentic interpreting practice.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the growth of interpreting technologies, from remote interpreting and Computer-Aided Interpreting to automated speech translation and interpreting avatars, there is now a high demand for ways to quickly and efficiently measure the quality of any interpreting delivered. A range of approaches to fulfil the need for quick and efficient quality measurement have been proposed, each involving some measure of automation. This article examines these recently-proposed quality measurement methods and will discuss their suitability for measuring the quality of authentic interpreting practice, whether delivered by humans or machines, concluding that automatic metrics as currently proposed cannot take into account the communicative context and thus are not viable measures of the quality of any interpreting provision when used on their own. Across all attempts to measure or even categorise quality in Interpreting Studies, the contexts in which interpreting takes place have become fundamental to the final analysis.

Related papers

• Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering [68.3400058037817]
  We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality.<n>We show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations.
  arXiv  Detail & Related papers  (2025-04-10T09:24:54Z)
• Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy [52.261323452286554]
  We introduce a method for contextual metric meta-evaluation by comparing the local metric accuracy of evaluation metrics.<n>Across translation, speech recognition, and ranking tasks, we demonstrate that the local metric accuracies vary both in absolute value and relative effectiveness as we shift across evaluation contexts.
  arXiv  Detail & Related papers  (2025-03-25T16:42:25Z)
• A Measure of the System Dependence of Automated Metrics [9.594167080604207]
  We argue that it is equally important to ensure that metrics treat all systems fairly and consistently.<n>In this paper, we introduce a method to evaluate this aspect.
  arXiv  Detail & Related papers  (2024-12-04T09:21:46Z)
• Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation [1.6982207802596105]
  This study investigates the convergences and divergences between automated metrics and human evaluation. To perform automatic assessment, four automated metrics are employed, while human evaluation incorporates the DQF-MQM error typology and six rubrics. Results underscore the indispensable role of human judgment in evaluating the performance of advanced translation tools.
  arXiv  Detail & Related papers  (2024-01-10T14:20:33Z)
• The Glass Ceiling of Automatic Evaluation in Natural Language Generation [60.59732704936083]
  We take a step back and analyze recent progress by comparing the body of existing automatic metrics and human metrics. Our extensive statistical analysis reveals surprising findings: automatic metrics -- old and new -- are much more similar to each other than to humans.
  arXiv  Detail & Related papers  (2022-08-31T01:13:46Z)
• Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
  This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
  arXiv  Detail & Related papers  (2021-11-15T12:09:08Z)
• Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods [9.210509295803243]
  We present a high-level and concise survey of translation quality assessment (TQA) methods, including both manual judgement criteria and automated evaluation metrics. We hope that this work will be an asset for both translation model researchers and quality assessment researchers.
  arXiv  Detail & Related papers  (2021-05-05T18:28:10Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)
• Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics [64.88815792555451]
  We show that current methods for judging metrics are highly sensitive to the translations used for assessment. We develop a method for thresholding performance improvement under an automatic metric against human judgements.
  arXiv  Detail & Related papers  (2020-06-11T09:12:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.