Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation

• URL: http://arxiv.org/abs/2410.07779v1
• Date: Thu, 10 Oct 2024 10:09:54 GMT
• Title: Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation
• Authors: Sweta Agrawal, José G. C. de Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno M Guerreiro, Andre Martins,
• Abstract summary: We propose an approach that leverages the best of both worlds. We first collect sentence-level quality assessments from professional linguists on translations generated by multiple high-quality MT systems. We then use this analysis to curate a new dataset, MT-Pref, which comprises 18k instances covering 18 language directions.
• Score: 18.077562738603792
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Alignment with human preferences is an important step in developing accurate and safe large language models. This is no exception in machine translation (MT), where better handling of language nuances and context-specific variations leads to improved quality. However, preference data based on human feedback can be very expensive to obtain and curate at a large scale. Automatic metrics, on the other hand, can induce preferences, but they might not match human expectations perfectly. In this paper, we propose an approach that leverages the best of both worlds. We first collect sentence-level quality assessments from professional linguists on translations generated by multiple high-quality MT systems and evaluate the ability of current automatic metrics to recover these preferences. We then use this analysis to curate a new dataset, MT-Pref (metric induced translation preference) dataset, which comprises 18k instances covering 18 language directions, using texts sourced from multiple domains post-2022. We show that aligning TOWER models on MT-Pref significantly improves translation quality on WMT23 and FLORES benchmarks.

Related papers

• Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis [20.023077870947024]
  This study focuses on Contrastive Preference Optimization (CPO) and conducts experiments to evaluate the impact of preference-based alignment on translation quality. Our findings indicate that while CPO consistently outperforms Supervised Fine-Tuning (SFT) on high-quality data with regard to the alignment metric, it may lead to instability across downstream evaluation metrics.
  arXiv  Detail & Related papers  (2024-09-30T08:01:44Z)
• Evaluating Automatic Metrics with Incremental Machine Translation Systems [55.78547133890403]
  We introduce a dataset comprising commercial machine translations, gathered weekly over six years across 12 translation directions. We assume commercial systems improve over time, which enables us to evaluate machine translation (MT) metrics based on their preference for more recent translations.
  arXiv  Detail & Related papers  (2024-07-03T17:04:17Z)
• Context-Aware Machine Translation with Source Coreference Explanation [26.336947440529713]
  We propose a model that explains the decisions made for translation by predicting coreference features in the input. We evaluate our method in the WMT document-level translation task of English-German dataset, the English-Russian dataset, and the multilingual TED talk dataset.
  arXiv  Detail & Related papers  (2024-04-30T12:41:00Z)
• Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains [10.743362634494842]
  We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on human-generated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to metrics that rely on the surface form, as well as pre-trained metrics which are not fine-tuned on MT quality judgements.
  arXiv  Detail & Related papers  (2024-02-28T23:01:24Z)
• Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution [57.42593422091653]
  We explore leveraging reinforcement learning with human feedback to improve translation quality. A reward model with strong language capabilities can more sensitively learn the subtle differences in translation quality.
  arXiv  Detail & Related papers  (2024-02-18T09:51:49Z)
• Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation [50.00235162432848]
  We train ALMA models with only 22K parallel sentences and 12M parameters. The resulting model, called ALMA-R, can match or exceed the performance of the WMT competition winners and GPT-4.
  arXiv  Detail & Related papers  (2024-01-16T15:04:51Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• Bring More Attention to Syntactic Symmetry for Automatic Postediting of High-Quality Machine Translations [4.217162744375792]
  We propose a linguistically motivated method of regularization that is expected to enhance APE models' understanding of the target language. Our analysis of experimental results demonstrates that the proposed method helps improving the state-of-the-art architecture's APE quality for high-quality MTs.
  arXiv  Detail & Related papers  (2023-05-17T20:25:19Z)
• Statistical Machine Translation for Indic Languages [1.8899300124593648]
  This paper canvasses about the development of bilingual Statistical Machine Translation models. To create the system, MOSES open-source SMT toolkit is explored. In our experiment, the quality of the translation is evaluated using standard metrics such as BLEU, METEOR, and RIBES.
  arXiv  Detail & Related papers  (2023-01-02T06:23:12Z)
• Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
  This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
  arXiv  Detail & Related papers  (2021-11-15T12:09:08Z)
• Machine Translation Customization via Automatic Training Data Selection from the Web [97.98885151955467]
  We describe an approach for customizing machine translation systems on specific domains. We select data similar to the target customer data to train neural translation models. Finally, we train MT models on our automatically selected data, obtaining a system specialized to the target domain.
  arXiv  Detail & Related papers  (2021-02-20T03:29:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.