MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew

• URL: http://arxiv.org/abs/2602.06546v1
• Date: Fri, 06 Feb 2026 09:51:45 GMT
• Title: MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew
• Abstract summary: MTQE.en-he is the first publicly available benchmark for Machine Translation Quality Estimation.<n>We benchmark ChatGPT prompting, TransQuest, and CometKiwi.<n>Fine-tuning experiments with TransQuest and CometKiwi reveal that full-model updates are sensitive to overfitting and distribution collapse.
• Score: 1.3821781249379796
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We release MTQE.en-he: to our knowledge, the first publicly available English-Hebrew benchmark for Machine Translation Quality Estimation. MTQE.en-he contains 959 English segments from WMT24++, each paired with a machine translation into Hebrew, and Direct Assessment scores of the translation quality annotated by three human experts. We benchmark ChatGPT prompting, TransQuest, and CometKiwi and show that ensembling the three models outperforms the best single model (CometKiwi) by 6.4 percentage points Pearson and 5.6 percentage points Spearman. Fine-tuning experiments with TransQuest and CometKiwi reveal that full-model updates are sensitive to overfitting and distribution collapse, yet parameter-efficient methods (LoRA, BitFit, and FTHead, i.e., fine-tuning only the classification head) train stably and yield improvements of 2-3 percentage points. MTQE.en-he and our experimental results enable future research on this under-resourced language pair.

Related papers

• Evaluating Automatic Metrics with Incremental Machine Translation Systems [55.78547133890403]
  We introduce a dataset comprising commercial machine translations, gathered weekly over six years across 12 translation directions. We assume commercial systems improve over time, which enables us to evaluate machine translation (MT) metrics based on their preference for more recent translations.
  arXiv  Detail & Related papers  (2024-07-03T17:04:17Z)
• Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean [7.843029855730508]
  We develop a 1200-sentence MQM evaluation benchmark for the language pair English-Korean. We find that reference-free setup outperforms its counterpart in the style dimension. Overall, RemBERT emerges as the most promising model.
  arXiv  Detail & Related papers  (2024-03-19T12:02:38Z)
• MTUncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning OpenAI LLMs [6.822926897514793]
  TQE is critical in assessing both machine translation (MT) and human translation (HT) quality without reference translations. This work examines whether the state-of-the-art large language models (LLMs) can be used for this purpose. We take OpenAI models as the best state-of-the-art technology and approach TQE as a binary classification task.
  arXiv  Detail & Related papers  (2023-07-31T21:13:30Z)
• Strategies for improving low resource speech to text translation relying on pre-trained ASR models [59.90106959717875]
  This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST) We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively.
  arXiv  Detail & Related papers  (2023-05-31T21:58:07Z)
• Evaluating and Improving the Coreference Capabilities of Machine Translation Models [30.60934078720647]
  Machine translation requires a wide range of linguistic capabilities. Current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora.
  arXiv  Detail & Related papers  (2023-02-16T18:16:09Z)
• Extrinsic Evaluation of Machine Translation Metrics [78.75776477562087]
  It is unclear if automatic metrics are reliable at distinguishing good translations from bad translations at the sentence level. We evaluate the segment-level performance of the most widely used MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of the downstream outcomes.
  arXiv  Detail & Related papers  (2022-12-20T14:39:58Z)
• M3ST: Mix at Three Levels for Speech Translation [66.71994367650461]
  We propose Mix at three levels for Speech Translation (M3ST) method to increase the diversity of the augmented training corpus. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. Experiments on MuST-C speech translation benchmark and analysis show that M3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
  arXiv  Detail & Related papers  (2022-12-07T14:22:00Z)
• Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
  This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
  arXiv  Detail & Related papers  (2021-11-15T12:09:08Z)
• ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback [70.5469946314539]
  ChrEnTranslate is an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability.
  arXiv  Detail & Related papers  (2021-07-30T17:58:54Z)
• Practical Perspectives on Quality Estimation for Machine Translation [6.400178956011897]
  Sentence level quality estimation (QE) for machine translation (MT) attempts to predict the translation edit rate (TER) cost of post-editing work required to correct MT output. We find consumers of MT output who are primarily interested in a binary quality metric: is the translated sentence adequate as-is or does it need post-editing? We demonstrate that, while classical QE regression models fare poorly on this task, they can be re-purposed by replacing the output regression layer with a binary classification one, achieving 50-60% recall at 90% precision.
  arXiv  Detail & Related papers  (2020-05-02T01:50:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.