Trainable Reference-Based Evaluation Metric for Identifying Quality of English-Gujarati Machine Translation System

• URL: http://arxiv.org/abs/2510.05113v1
• Date: Sat, 27 Sep 2025 16:05:15 GMT
• Title: Trainable Reference-Based Evaluation Metric for Identifying Quality of English-Gujarati Machine Translation System
• Abstract summary: We have introduced a reference-based MT evaluation metric for Gujarati.<n>We have trained two versions of the metric which uses 25 features for training.<n>It was found that the metrics produced better human correlations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Machine Translation (MT) Evaluation is an integral part of the MT development life cycle. Without analyzing the outputs of MT engines, it is impossible to evaluate the performance of an MT system. Through experiments, it has been identified that what works for English and other European languages does not work well with Indian languages. Thus, In this paper, we have introduced a reference-based MT evaluation metric for Gujarati which is based on supervised learning. We have trained two versions of the metric which uses 25 features for training. Among the two models, one model is trained using 6 hidden layers with 500 epochs while the other model is trained using 10 hidden layers with 500 epochs. To test the performance of the metric, we collected 1000 MT outputs of seven MT systems. These MT engine outputs were compared with 1 human reference translation. While comparing the developed metrics with other available metrics, it was found that the metrics produced better human correlations.

Related papers

• Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation [57.11989521509119]
  We propose a novel agentic translation evaluation framework, centered by a reflective Core Agent that invokes specialized sub-agents.<n> Experimental results indicate the efficacy of RATE, achieving an improvement of at least 3.2 meta score compared with current metrics.
  arXiv  Detail & Related papers  (2026-01-12T09:03:42Z)
• Retrieval-Augmented Machine Translation with Unstructured Knowledge [63.97706326080482]
  Retrieval-augmented generation (RAG) introduces additional information to enhance large language models (LLMs)<n>In machine translation (MT), previous work typically retrieves in-context examples from paired MT corpora, or domain-specific knowledge from knowledge graphs.<n>In this paper, we study retrieval-augmented MT using unstructured documents.
  arXiv  Detail & Related papers  (2024-12-05T17:00:32Z)
• Evaluating Automatic Metrics with Incremental Machine Translation Systems [55.78547133890403]
  We introduce a dataset comprising commercial machine translations, gathered weekly over six years across 12 translation directions. We assume commercial systems improve over time, which enables us to evaluate machine translation (MT) metrics based on their preference for more recent translations.
  arXiv  Detail & Related papers  (2024-07-03T17:04:17Z)
• Trained MT Metrics Learn to Cope with Machine-translated References [47.00411750716812]
  We show that Prism+FT becomes more robust to machine-translated references. This suggests that the effects of metric training go beyond the intended effect of improving overall correlation with human judgments.
  arXiv  Detail & Related papers  (2023-12-01T12:15:58Z)
• Extrinsic Evaluation of Machine Translation Metrics [78.75776477562087]
  It is unclear if automatic metrics are reliable at distinguishing good translations from bad translations at the sentence level. We evaluate the segment-level performance of the most widely used MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of the downstream outcomes.
  arXiv  Detail & Related papers  (2022-12-20T14:39:58Z)
• T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics [94.69907794006826]
  We present a framework that combines the best of both worlds, using both supervised and unsupervised signals from whatever data we have available. We operationalize this idea by training T5Score, a metric that uses these training signals with mT5 as the backbone. T5Score achieves the best performance on all datasets against existing top-scoring metrics at the segment level.
  arXiv  Detail & Related papers  (2022-12-12T06:29:04Z)
• Difficulty-Aware Machine Translation Evaluation [19.973201669851626]
  We propose a novel difficulty-aware machine translation evaluation metric. A translation that fails to be predicted by most MT systems will be treated as a difficult one and assigned a large weight in the final score function. Our proposed method performs well even when all the MT systems are very competitive.
  arXiv  Detail & Related papers  (2021-07-30T02:45:36Z)
• Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing [11.564158965143418]
  We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser. We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task. Our method is simple and intuitive, and does not require human judgements for training.
  arXiv  Detail & Related papers  (2020-04-30T03:32:34Z)
• Meta-Evaluation of Translation Evaluation Methods: a systematic up-to-date overview [4.880243880711164]
  Starting from the 1950s, Machine Translation (MT) was challenged by different scientific solutions.<n>In this tutorial, we overview the traditional human judgement criteria, automatic evaluation metrics, and the meta-evaluation of the evaluation methods.
  arXiv  Detail & Related papers  (2016-05-15T09:41:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.