A Bayesian approach to translators' reliability assessment

• URL: http://arxiv.org/abs/2203.07135v1
• Date: Mon, 14 Mar 2022 14:29:45 GMT
• Title: A Bayesian approach to translators' reliability assessment
• Authors: Marco Miccheli, Andrea Tacchella, Andrea Zaccaria, Dario Mazzilli, S\'ebastien Brati\`eres, Luciano Pietronero
• Abstract summary: We consider the Translation Quality Assessment process as a complex process, considering it from the physics of complex systems point of view. We build two Bayesian models that parameterise the features involved in the TQA process, namely the translation difficulty, the characteristics of the translators involved in producing the translation and assessing its quality. We show that reviewers reliability cannot be taken for granted even if they are expert translators.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Translation Quality Assessment (TQA) conducted by human translators is a process widely used, both in estimating the increasingly used Machine Translation performance and in finding an agreement between customers and translation providers in translation industry. While translation scholars are aware about the importance of having a reliable way to conduct the TQA process, it seems that there is limited literature facing the issue of reliability with a quantitative approach. Here we consider the TQA as a complex process, considering it from the physics of complex systems point of view, and we face the reliability issue with a Bayesian approach. Using a dataset of translation quality evaluations, in an error annotation setting, entirely produced by the Language Service Provider Translated Srl, we build two Bayesian models that parameterise the features involved in the TQA process, namely the translation difficulty, the characteristics of the translators involved in producing the translation and assessing its quality (reviewers). After validating the models in an unsupervised setting, showing that it is possible to get meaningful insights about translators even with just one review per translation job, we extract information about the translators and reviewers and we show that reviewers reliability cannot be taken for granted even if they are expert translators: the translator's expertise could induce also a cognitive bias when reviewing a translation produced by another translator. The most expert translators, though, show the highest level of consistency, both in the task of translating and in the one of assessing translation quality.

Related papers

• xTower: A Multilingual LLM for Explaining and Correcting Translation Errors [22.376508000237042]
  xTower is an open large language model (LLM) built on top of TowerBase to provide free-text explanations for translation errors. We test xTower across various experimental setups in generating translation corrections, demonstrating significant improvements in translation quality.
  arXiv  Detail & Related papers  (2024-06-27T18:51:46Z)
• Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective [72.83966378613238]
  Under-translation and over-translation remain two challenging problems in state-of-the-art Neural Machine Translation (NMT) systems. We conduct an in-depth analysis on the underlying cause of under-translation in NMT, providing an explanation from the perspective of decoding objective. We propose employing the confidence of predicting End Of Sentence (EOS) as a detector for under-translation, and strengthening the confidence-based penalty to penalize candidates with a high risk of under-translation.
  arXiv  Detail & Related papers  (2024-05-29T09:25:49Z)
• Evaluating Optimal Reference Translations [4.956416618428049]
  We propose a methodology for creating more reliable document-level human reference translations. We evaluate the obtained document-level optimal reference translations in comparison with "standard" ones.
  arXiv  Detail & Related papers  (2023-11-28T13:50:50Z)
• Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT's Customizability [0.0]
  The study reveals that the inclusion of suitable prompts in large-scale language models like ChatGPT can yield flexible translations. The research scrutinizes the changes in translation quality when prompts are used to generate translations that meet specific conditions.
  arXiv  Detail & Related papers  (2023-08-02T19:11:04Z)
• Extrinsic Evaluation of Machine Translation Metrics [78.75776477562087]
  It is unclear if automatic metrics are reliable at distinguishing good translations from bad translations at the sentence level. We evaluate the segment-level performance of the most widely used MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of the downstream outcomes.
  arXiv  Detail & Related papers  (2022-12-20T14:39:58Z)
• Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? [61.866103154161884]
  Neural machine translation (NMT) is often criticized for failures that happen without awareness. We propose a novel competency-aware NMT by extending conventional NMT with a self-estimator. We show that the proposed method delivers outstanding performance on quality estimation.
  arXiv  Detail & Related papers  (2022-11-25T02:39:41Z)
• Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
  This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
  arXiv  Detail & Related papers  (2021-11-15T12:09:08Z)
• ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback [70.5469946314539]
  ChrEnTranslate is an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability.
  arXiv  Detail & Related papers  (2021-07-30T17:58:54Z)
• Backtranslation Feedback Improves User Confidence in MT, Not Quality [18.282199360280433]
  We show three ways in which user confidence in the outbound translation, as well as its overall final quality, can be affected. In this paper, we describe an experiment on outbound translation from English to Czech and Estonian. We show that backward translation feedback has a mixed effect on the whole process: it increases user confidence in the produced translation, but not the objective quality.
  arXiv  Detail & Related papers  (2021-04-12T17:50:24Z)
• A Set of Recommendations for Assessing Human-Machine Parity in Language Translation [87.72302201375847]
  We reassess Hassan et al.'s investigation into Chinese to English news translation. We show that the professional human translations contained significantly fewer errors.
  arXiv  Detail & Related papers  (2020-04-03T17:49:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.