Verdi: Quality Estimation and Error Detection for Bilingual
- URL: http://arxiv.org/abs/2105.14878v1
- Date: Mon, 31 May 2021 11:04:13 GMT
- Title: Verdi: Quality Estimation and Error Detection for Bilingual
- Authors: Mingjun Zhao, Haijiang Wu, Di Niu, Zixuan Wang, Xiaoli Wang
- Abstract summary: Verdi is a novel framework for word-level and sentence-level post-editing effort estimation for bilingual corpora.
We exploit the symmetric nature of bilingual corpora and apply model-level dual learning in the NMT predictor.
Our method beats the winner of the competition and outperforms other baseline methods by a great margin.
- Score: 23.485380293716272
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Translation Quality Estimation is critical to reducing post-editing efforts
in machine translation and to cross-lingual corpus cleaning. As a research
problem, quality estimation (QE) aims to directly estimate the quality of
translation in a given pair of source and target sentences, and highlight the
words that need corrections, without referencing to golden translations. In
this paper, we propose Verdi, a novel framework for word-level and
sentence-level post-editing effort estimation for bilingual corpora. Verdi
adopts two word predictors to enable diverse features to be extracted from a
pair of sentences for subsequent quality estimation, including a
transformer-based neural machine translation (NMT) model and a pre-trained
cross-lingual language model (XLM). We exploit the symmetric nature of
bilingual corpora and apply model-level dual learning in the NMT predictor,
which handles a primal task and a dual task simultaneously with weight sharing,
leading to stronger context prediction ability than single-direction NMT
models. By taking advantage of the dual learning scheme, we further design a
novel feature to directly encode the translated target information without
relying on the source context. Extensive experiments conducted on WMT20 QE
tasks demonstrate that our method beats the winner of the competition and
outperforms other baseline methods by a great margin. We further use the
sentence-level scores provided by Verdi to clean a parallel corpus and observe
benefits on both model performance and training efficiency.
Related papers
- BiVert: Bidirectional Vocabulary Evaluation using Relations for Machine
Translation [4.651581292181871]
We propose a bidirectional semantic-based evaluation method designed to assess the sense distance of the translation from the source text.
This approach employs the comprehensive multilingual encyclopedic dictionary BabelNet.
Factual analysis shows a strong correlation between the average evaluation scores generated by our method and the human assessments across various machine translation systems for English-German language pair.
arXiv Detail & Related papers (2024-03-06T08:02:21Z) - Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text.
The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
arXiv Detail & Related papers (2021-11-15T12:09:08Z) - Improving Multilingual Translation by Representation and Gradient
Regularization [82.42760103045083]
We propose a joint approach to regularize NMT models at both representation-level and gradient-level.
Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance.
arXiv Detail & Related papers (2021-09-10T10:52:21Z) - Distributionally Robust Multilingual Machine Translation [94.51866646879337]
We propose a new learning objective for Multilingual neural machine translation (MNMT) based on distributionally robust optimization.
We show how to practically optimize this objective for large translation corpora using an iterated best response scheme.
Our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.
arXiv Detail & Related papers (2021-09-09T03:48:35Z) - Ensemble Fine-tuned mBERT for Translation Quality Estimation [0.0]
In this paper, we discuss our submission to the WMT 2021 QE Shared Task.
Our proposed system is an ensemble of multilingual BERT (mBERT)-based regression models.
It demonstrates comparable performance with respect to the Pearson's correlation and beats the baseline system in MAE/ RMSE for several language pairs.
arXiv Detail & Related papers (2021-09-08T20:13:06Z) - Modelling Latent Translations for Cross-Lingual Transfer [47.61502999819699]
We propose a new technique that integrates both steps of the traditional pipeline (translation and classification) into a single model.
We evaluate our novel latent translation-based model on a series of multilingual NLU tasks.
We report gains for both zero-shot and few-shot learning setups, up to 2.7 accuracy points on average.
arXiv Detail & Related papers (2021-07-23T17:11:27Z) - On the Language Coverage Bias for Neural Machine Translation [81.81456880770762]
Language coverage bias is important for neural machine translation (NMT) because the target-original training data is not well exploited in current practice.
By carefully designing experiments, we provide comprehensive analyses of the language coverage bias in the training data.
We propose two simple and effective approaches to alleviate the language coverage bias problem.
arXiv Detail & Related papers (2021-06-07T01:55:34Z) - Learning Contextualized Sentence Representations for Document-Level
Neural Machine Translation [59.191079800436114]
Document-level machine translation incorporates inter-sentential dependencies into the translation of a source sentence.
We propose a new framework to model cross-sentence dependencies by training neural machine translation (NMT) to predict both the target translation and surrounding sentences of a source sentence.
arXiv Detail & Related papers (2020-03-30T03:38:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.