Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation

• URL: http://arxiv.org/abs/2608.03577v1
• Date: Tue, 04 Aug 2026 12:33:13 GMT
• Title: Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation
• Abstract summary: We argue that current QE systems are structurally ill-equipped to serve as reliable standalone tools in real-world translation.<n>The evaluation of the quality of translation at the level of isolated segments is problematic because it tends to miss out on cohesion, coherence, and stylistic and rhetorical text features.
• Score: 3.0039231903051875
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Automation of Translation Quality Estimation (QE) has emerged as a widely discussed approach to managing translation quality at scale, and a growing number of tools and technologies have been released in pursuit of this goal. However, the proliferation of new QE systems has not always been accompanied by robust, transparent, and reproducible research and testing. This gap deserves critical scrutiny. This paper examines some fundamental limitations of the QE technology from both theoretical and empirical perspectives, arguing that current QE systems are structurally ill-equipped to serve as reliable standalone tools in real-world translation workflows. The reviewed evidence suggests that QE suffers from a range of interrelated and largely unresolved limitations. Most fundamentally, the evaluation of the quality of translation at the level of isolated segments is problematic because it tends to miss out on cohesion, coherence, and stylistic and rhetorical text features. In addition, empirical research documents several other limitations and flaws, including failure to generalize, systematic biases, overfitting and distribution collapse, performance gaps, error annotation challenges, and data scarcity. These are structural limitations arising from the complexity of human language and translation as a cognitive and communicative act - limitations that more data and better architectures have so far not overcome. Consequently, segment-level QE scores should not be used as a standalone basis for routing, release, or review bypass in production; we argue future work should focus on automating human evaluation grounded in MQM.

Related papers

• IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment [59.83941105390897]
  We propose IQA-T1, a tool-based visual evidence reasoning framework.<n>During inference, the model autonomously invokes specialized analysis tools to generate structured visual evidence.<n>Experiments show that IQA-T1 achieves the best overall performance across datasets.
  arXiv  Detail & Related papers  (2026-07-14T05:47:32Z)
• Semi-Synthetic Parallel Data for Translation Quality Estimation: A Case Study of Dataset Building for an Under-Resourced Language Pair [0.0]
  This study presents a semi-synthetic parallel dataset for English-to-Hebrew QE.<n>We incorporated professionally translated English-Hebrew segments from our own resources, which were assigned the highest quality score.<n> Controlled translation errors were introduced to address linguistic challenges, particularly regarding gender and number agreement.
  arXiv  Detail & Related papers  (2026-03-12T09:48:34Z)
• Beyond Scalar Scores: Reinforcement Learning for Error-Aware Quality Estimation of Machine Translation [10.050982803590903]
  Quality Estimation aims to assess the quality of machine translation (MT) outputs without relying on reference translations.<n>We introduce the first segment-level QE dataset for English to Malayalam, a severely resource-scarce language pair.<n>We introduce ALOPE-RL, a policy-based reinforcement learning framework that trains efficient adapters.
  arXiv  Detail & Related papers  (2026-02-09T12:42:41Z)
• Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models [4.155649113742267]
  ReQUESTA is a hybrid, multi-agent framework for generating cognitively diverse multiple-choice questions (MCQs)<n>We evaluated the framework in a large-scale reading comprehension study using academic expository texts.<n>Results showed that ReQUESTA-generated items were consistently more challenging, more discriminative, and more strongly aligned with overall reading comprehension performance.
  arXiv  Detail & Related papers  (2026-02-03T16:26:47Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Federated In-Context Learning: Iterative Refinement for Improved Answer Quality [62.72381208029899]
  In-context learning (ICL) enables language models to generate responses without modifying their parameters by leveraging examples provided in the input.<n>We propose Federated In-Context Learning (Fed-ICL), a general framework that enhances ICL through an iterative, collaborative process.<n>Fed-ICL progressively refines responses by leveraging multi-round interactions between clients and a central server, improving answer quality without the need to transmit model parameters.
  arXiv  Detail & Related papers  (2025-06-09T05:33:28Z)
• Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering [68.3400058037817]
  We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality.<n>We show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations.
  arXiv  Detail & Related papers  (2025-04-10T09:24:54Z)
• Perturbation-based QE: An Explainable, Unsupervised Word-level Quality Estimation Method for Blackbox Machine Translation [12.376309678270275]
  Perturbation-based QE works simply by analyzing MT system output on perturbed input source sentences. Our approach is better at detecting gender bias and word-sense-disambiguation errors in translation than supervised QE.
  arXiv  Detail & Related papers  (2023-05-12T13:10:57Z)
• Shortcomings of Question Answering Based Factuality Frameworks for Error Localization [51.01957350348377]
  We show that question answering (QA)-based factuality metrics fail to correctly identify error spans in generated summaries. Our analysis reveals a major reason for such poor localization: questions generated by the QG module often inherit errors from non-factual summaries which are then propagated further into downstream modules. Our experiments conclusively show that there exist fundamental issues with localization using the QA framework which cannot be fixed solely by stronger QA and QG models.
  arXiv  Detail & Related papers  (2022-10-13T05:23:38Z)
• Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
  This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
  arXiv  Detail & Related papers  (2021-11-15T12:09:08Z)
• Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation [25.325624543852086]
  We propose a general methodology for adversarial testing of Quality Estimation for Machine Translation (MT) systems. We show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaning-preserving and meaning-altering perturbations is predictive of its overall performance.
  arXiv  Detail & Related papers  (2021-09-22T17:32:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.