Translation Quality Assessment: A Brief Survey on Manual and Automatic
Methods
- URL: http://arxiv.org/abs/2105.03311v1
- Date: Wed, 5 May 2021 18:28:10 GMT
- Title: Translation Quality Assessment: A Brief Survey on Manual and Automatic
Methods
- Authors: Lifeng Han, Gareth J. F. Jones and Alan F. Smeaton
- Abstract summary: We present a high-level and concise survey of translation quality assessment (TQA) methods, including both manual judgement criteria and automated evaluation metrics.
We hope that this work will be an asset for both translation model researchers and quality assessment researchers.
- Score: 9.210509295803243
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: To facilitate effective translation modeling and translation studies, one of
the crucial questions to address is how to assess translation quality. From the
perspectives of accuracy, reliability, repeatability and cost, translation
quality assessment (TQA) itself is a rich and challenging task. In this work,
we present a high-level and concise survey of TQA methods, including both
manual judgement criteria and automated evaluation metrics, which we classify
into further detailed sub-categories. We hope that this work will be an asset
for both translation model researchers and quality assessment researchers. In
addition, we hope that it will enable practitioners to quickly develop a better
understanding of the conventional TQA field, and to find corresponding closely
relevant evaluation solutions for their own needs. This work may also serve
inspire further development of quality assessment and evaluation methodologies
for other natural language processing (NLP) tasks in addition to machine
translation (MT), such as automatic text summarization (ATS), natural language
understanding (NLU) and natural language generation (NLG).
Related papers
- PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models [72.57329554067195]
ProxyQA is an innovative framework dedicated to assessing longtext generation.
It comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers.
It assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions.
arXiv Detail & Related papers (2024-01-26T18:12:25Z) - Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation [64.5862977630713]
This study investigates how Large Language Models (LLMs) leverage source and reference data in machine translation evaluation task.
We find that reference information significantly enhances the evaluation accuracy, while surprisingly, source information sometimes is counterproductive.
arXiv Detail & Related papers (2024-01-12T13:23:21Z) - Competency-Aware Neural Machine Translation: Can Machine Translation
Know its Own Translation Quality? [61.866103154161884]
Neural machine translation (NMT) is often criticized for failures that happen without awareness.
We propose a novel competency-aware NMT by extending conventional NMT with a self-estimator.
We show that the proposed method delivers outstanding performance on quality estimation.
arXiv Detail & Related papers (2022-11-25T02:39:41Z) - PreQuEL: Quality Estimation of Machine Translation Outputs in Advance [32.922128367314194]
A PreQuEL system predicts how well a given sentence will be translated, without recourse to the actual translation.
We develop a baseline model for the task and analyze its performance.
We show that this augmentation method can improve the performance of the Quality-Estimation task as well.
arXiv Detail & Related papers (2022-05-18T18:55:05Z) - A Bayesian approach to translators' reliability assessment [0.0]
We consider the Translation Quality Assessment process as a complex process, considering it from the physics of complex systems point of view.
We build two Bayesian models that parameterise the features involved in the TQA process, namely the translation difficulty, the characteristics of the translators involved in producing the translation and assessing its quality.
We show that reviewers reliability cannot be taken for granted even if they are expert translators.
arXiv Detail & Related papers (2022-03-14T14:29:45Z) - An Overview on Machine Translation Evaluation [6.85316573653194]
Machine translation (MT) has become one of the important tasks of AI and development.
The evaluation task of MT is not only to evaluate the quality of machine translation, but also to give timely feedback to machine translation researchers.
This report mainly includes a brief history of machine translation evaluation (MTE), the classification of research methods on MTE, and the the cutting-edge progress.
arXiv Detail & Related papers (2022-02-22T16:58:28Z) - QEMind: Alibaba's Submission to the WMT21 Quality Estimation Shared Task [24.668012925628968]
We present our submissions to the WMT 2021 QE shared task.
We propose several useful features to evaluate the uncertainty of the translations to build our QE system, named textitQEMind.
We show that our multilingual systems outperform the best system in the Direct Assessment QE task of WMT 2020.
arXiv Detail & Related papers (2021-12-30T02:27:29Z) - Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text.
The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
arXiv Detail & Related papers (2021-11-15T12:09:08Z) - TextFlint: Unified Multilingual Robustness Evaluation Toolkit for
Natural Language Processing [73.16475763422446]
We propose a multilingual robustness evaluation platform for NLP tasks (TextFlint)
It incorporates universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analysis.
TextFlint generates complete analytical reports as well as targeted augmented data to address the shortcomings of the model's robustness.
arXiv Detail & Related papers (2021-03-21T17:20:38Z) - Unsupervised Quality Estimation for Neural Machine Translation [63.38918378182266]
Existing approaches require large amounts of expert annotated data, computation and time for training.
We devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required.
We achieve very good correlation with human judgments of quality, rivalling state-of-the-art supervised QE models.
arXiv Detail & Related papers (2020-05-21T12:38:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.