MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation

• URL: http://arxiv.org/abs/2412.11615v1
• Date: Mon, 16 Dec 2024 09:57:28 GMT
• Title: MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation
• Authors: Javier García Gilabert, Carlos Escolano, Audrey Mash, Xixian Liao, Maite Melero,
• Abstract summary: MT-LENS is a framework designed to evaluate Machine Translation (MT) systems across a variety of tasks. It offers a user-friendly platform to compare systems and analyze translations with interactive visualizations.
• Score: 1.7775825387442485
• License:
• Abstract: We introduce MT-LENS, a framework designed to evaluate Machine Translation (MT) systems across a variety of tasks, including translation quality, gender bias detection, added toxicity, and robustness to misspellings. While several toolkits have become very popular for benchmarking the capabilities of Large Language Models (LLMs), existing evaluation tools often lack the ability to thoroughly assess the diverse aspects of MT performance. MT-LENS addresses these limitations by extending the capabilities of LM-eval-harness for MT, supporting state-of-the-art datasets and a wide range of evaluation metrics. It also offers a user-friendly platform to compare systems and analyze translations with interactive visualizations. MT-LENS aims to broaden access to evaluation strategies that go beyond traditional translation quality evaluation, enabling researchers and engineers to better understand the performance of a NMT model and also easily measure system's biases.

Related papers

• Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics [46.71836180414362]
  We introduce an interpretable evaluation framework for Machine Translation (MT) metrics. Within this framework, we evaluate metrics in two scenarios that serve as proxies for the data filtering and translation re-ranking use cases. We also raise concerns regarding the reliability of manually curated data following the Direct Assessments+Scalar Quality Metrics (DA+SQM) guidelines.
  arXiv  Detail & Related papers  (2024-10-07T16:42:10Z)
• Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding [73.32763904267186]
  Large Language Models (LLMs) present the potential for achieving superior translation quality. We propose Cooperative Decoding (CoDec) which treats NMT systems as a pretranslation model and MT-oriented LLMs as a supplemental solution.
  arXiv  Detail & Related papers  (2023-11-06T03:41:57Z)
• Automating Behavioral Testing in Machine Translation [9.151054827967933]
  We propose to use Large Language Models to generate source sentences tailored to test the behavior of Machine Translation models. We can then verify whether the MT model exhibits the expected behavior through matching candidate sets. Our approach aims to make behavioral testing of MT systems practical while requiring only minimal human effort.
  arXiv  Detail & Related papers  (2023-09-05T19:40:45Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training [64.37683359609308]
  In this study, we analyze various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. We find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm.
  arXiv  Detail & Related papers  (2023-07-06T16:59:30Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• Machine Translation Impact in E-commerce Multilingual Search [0.0]
  Cross-lingual information retrieval correlates highly with the quality of Machine Translation. There may be a threshold beyond which improving query translation quality yields little or no benefit to further improve the retrieval performance.
  arXiv  Detail & Related papers  (2023-01-31T21:59:35Z)
• Extrinsic Evaluation of Machine Translation Metrics [78.75776477562087]
  It is unclear if automatic metrics are reliable at distinguishing good translations from bad translations at the sentence level. We evaluate the segment-level performance of the most widely used MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of the downstream outcomes.
  arXiv  Detail & Related papers  (2022-12-20T14:39:58Z)
• Prompting PaLM for Translation: Assessing Strategies and Performance [16.73524055296411]
  pathways language model (PaLM) has demonstrated the strongest machine translation (MT) performance among similarly-trained LLMs to date. We revisit previous assessments of PaLM's MT capabilities with more recent test sets, modern MT metrics, and human evaluation, and find that its performance, while impressive, still lags that of state-of-the-art supervised systems.
  arXiv  Detail & Related papers  (2022-11-16T18:42:37Z)
• HilMeMe: A Human-in-the-Loop Machine Translation Evaluation Metric Looking into Multi-Word Expressions [6.85316573653194]
  We describe the design and implementation of a linguistically motivated human-in-the-loop evaluation metric looking into idiomatic and terminological Multi-word Expressions (MWEs) MWEs can be used as one of the main factors to distinguish different MT systems by looking into their capabilities in recognising and translating MWEs in an accurate and meaning equivalent manner.
  arXiv  Detail & Related papers  (2022-11-09T21:15:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.