Ensemble Fine-tuned mBERT for Translation Quality Estimation

• URL: http://arxiv.org/abs/2109.03914v1
• Date: Wed, 8 Sep 2021 20:13:06 GMT
• Title: Ensemble Fine-tuned mBERT for Translation Quality Estimation
• Authors: Shaika Chowdhury, Naouel Baili and Brian Vannah
• Abstract summary: In this paper, we discuss our submission to the WMT 2021 QE Shared Task. Our proposed system is an ensemble of multilingual BERT (mBERT)-based regression models. It demonstrates comparable performance with respect to the Pearson's correlation and beats the baseline system in MAE/ RMSE for several language pairs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quality Estimation (QE) is an important component of the machine translation workflow as it assesses the quality of the translated output without consulting reference translations. In this paper, we discuss our submission to the WMT 2021 QE Shared Task. We participate in Task 2 sentence-level sub-task that challenge participants to predict the HTER score for sentence-level post-editing effort. Our proposed system is an ensemble of multilingual BERT (mBERT)-based regression models, which are generated by fine-tuning on different input settings. It demonstrates comparable performance with respect to the Pearson's correlation and beats the baseline system in MAE/ RMSE for several language pairs. In addition, we adapt our system for the zero-shot setting by exploiting target language-relevant language pairs and pseudo-reference translations.

Related papers

• Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation [0.6998085564793366]
  This work introduces QE-fusion, a method that synthesizes translations using a quality estimation metric (QE) We demonstrate that our approach generates novel translations in over half of the cases. We empirically establish that QE-fusion scales linearly with the number of candidates in the pool.
  arXiv  Detail & Related papers  (2024-01-12T16:52:41Z)
• Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task [59.46906545506715]
  We introduce the NJUNLP team to the WMT 2023 Quality Estimation (QE) shared task. Our team submitted predictions for the English-German language pair on all two sub-tasks. Our models achieved the best results in English-German for both word-level and fine-grained error span detection sub-tasks.
  arXiv  Detail & Related papers  (2023-09-23T01:52:14Z)
• Extrinsic Evaluation of Machine Translation Metrics [78.75776477562087]
  It is unclear if automatic metrics are reliable at distinguishing good translations from bad translations at the sentence level. We evaluate the segment-level performance of the most widely used MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of the downstream outcomes.
  arXiv  Detail & Related papers  (2022-12-20T14:39:58Z)
• Alibaba-Translate China's Submission for WMT 2022 Quality Estimation Shared Task [80.22825549235556]
  We present our submission to the sentence-level MQM benchmark at Quality Estimation Shared Task, named UniTE. Specifically, our systems employ the framework of UniTE, which combined three types of input formats during training with a pre-trained language model. Results show that our models reach 1st overall ranking in the Multilingual and English-Russian settings, and 2nd overall ranking in English-German and Chinese-English settings.
  arXiv  Detail & Related papers  (2022-10-18T08:55:27Z)
• QEMind: Alibaba's Submission to the WMT21 Quality Estimation Shared Task [24.668012925628968]
  We present our submissions to the WMT 2021 QE shared task. We propose several useful features to evaluate the uncertainty of the translations to build our QE system, named textitQEMind. We show that our multilingual systems outperform the best system in the Direct Assessment QE task of WMT 2020.
  arXiv  Detail & Related papers  (2021-12-30T02:27:29Z)
• On Cross-Lingual Retrieval with Multilingual Text Encoders [51.60862829942932]
  We study the suitability of state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. We benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR experiments. We evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments.
  arXiv  Detail & Related papers  (2021-12-21T08:10:27Z)
• Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
  This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
  arXiv  Detail & Related papers  (2021-11-15T12:09:08Z)
• Verdi: Quality Estimation and Error Detection for Bilingual [23.485380293716272]
  Verdi is a novel framework for word-level and sentence-level post-editing effort estimation for bilingual corpora. We exploit the symmetric nature of bilingual corpora and apply model-level dual learning in the NMT predictor. Our method beats the winner of the competition and outperforms other baseline methods by a great margin.
  arXiv  Detail & Related papers  (2021-05-31T11:04:13Z)
• Ensemble-based Transfer Learning for Low-resource Machine Translation Quality Estimation [1.7188280334580195]
  We focus on the Sentence-Level QE Shared Task of the Fifth Conference on Machine Translation (WMT20) We propose an ensemble-based predictor-estimator QE model with transfer learning to overcome such QE data scarcity challenge. We achieve the best performance on the ensemble model combining the models pretrained by individual languages as well as different levels of parallel trained corpus with a Pearson's correlation of 0.298.
  arXiv  Detail & Related papers  (2021-05-17T06:02:17Z)
• Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task [30.889496911261677]
  This paper describes our contribution to the WMT 2020 Metrics Shared Task. We make several submissions based on BLEURT, a metric based on transfer learning. We show how to combine BLEURT's predictions with those of YiSi and use alternative reference translations to enhance the performance.
  arXiv  Detail & Related papers  (2020-10-08T23:16:26Z)
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation [55.02832094101173]
  Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual similarity. This paper concerns ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations.
  arXiv  Detail & Related papers  (2020-05-03T22:10:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.