Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translation

• URL: http://arxiv.org/abs/2405.11937v1
• Date: Mon, 20 May 2024 10:25:03 GMT
• Title: Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translation
• Authors: Kamil Guttmann, Mikołaj Pokrywka, Adrian Charkiewicz, Artur Nowakowski,
• Abstract summary: This paper explores Minimum Bayes Risk (MBR) decoding for self-improvement in machine translation (MT) We implement the self-improvement process by fine-tuning the model on its MBR-decoded forward translations. The results demonstrate significant enhancements in translation quality for all examined language pairs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper explores Minimum Bayes Risk (MBR) decoding for self-improvement in machine translation (MT), particularly for domain adaptation and low-resource languages. We implement the self-improvement process by fine-tuning the model on its MBR-decoded forward translations. By employing COMET as the MBR utility metric, we aim to achieve the reranking of translations that better aligns with human preferences. The paper explores the iterative application of this approach and the potential need for language-specific MBR utility metrics. The results demonstrate significant enhancements in translation quality for all examined language pairs, including successful application to domain-adapted models and generalisation to low-resource settings. This highlights the potential of COMET-guided MBR for efficient MT self-improvement in various scenarios.

Related papers

• Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine Translation [30.323103270892734]
  Maximum a posteriori decoding, a commonly used method for neural machine translation (NMT), aims to maximize the estimated posterior probability. Minimum Bayes Risk (MBR) decoding offers an alternative by seeking hypotheses with the highest expected utility.
  arXiv  Detail & Related papers  (2024-06-17T15:13:52Z)
• Enhancing Neural Machine Translation of Low-Resource Languages: Corpus Development, Human Evaluation and Explainable AI Architectures [0.0]
  The Transformer architecture stands out as the gold standard, especially for high-resource language pairs. The scarcity of parallel datasets for low-resource languages can hinder machine translation development. This thesis introduces adaptNMT and adaptMLLM, two open-source applications streamlined for the development, fine-tuning, and deployment of neural machine translation models.
  arXiv  Detail & Related papers  (2024-03-03T18:08:30Z)
• Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding [15.309135455863753]
  We show how the recently developed Reinforcement Learning technique, Direct Preference Optimization (DPO), can fine-tune Multilingual Large Language Models without additional computation. Our method uses only a small monolingual fine-tuning set and yields significantly improved performance on multiple NMT test sets compared to MLLMs without DPO.
  arXiv  Detail & Related papers  (2023-11-14T18:43:51Z)
• Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding [73.32763904267186]
  Large Language Models (LLMs) present the potential for achieving superior translation quality. We propose Cooperative Decoding (CoDec) which treats NMT systems as a pretranslation model and MT-oriented LLMs as a supplemental solution.
  arXiv  Detail & Related papers  (2023-11-06T03:41:57Z)
• Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Model [77.19693792957614]
  We propose to make neural machine translation (NMT) models quality-aware by training them to estimate the quality of their own output. We obtain quality gains similar or even superior to quality reranking approaches, but with the efficiency of single pass decoding.
  arXiv  Detail & Related papers  (2023-10-10T15:33:51Z)
• Improving Translation Faithfulness of Large Language Models via Augmenting Instructions [89.76691340615848]
  We propose SWIE (Segment-Weighted Instruction Embedding) and an instruction-following dataset OVERMISS. SWIE improves the model instruction understanding by adding a global instruction representation on the following input and response representations. OVERMISS improves model faithfulness by comparing over-translation and miss-translation results with the correct translation.
  arXiv  Detail & Related papers  (2023-08-24T09:32:29Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training [64.37683359609308]
  In this study, we analyze various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. We find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm.
  arXiv  Detail & Related papers  (2023-07-06T16:59:30Z)
• Strategies for improving low resource speech to text translation relying on pre-trained ASR models [59.90106959717875]
  This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST) We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively.
  arXiv  Detail & Related papers  (2023-05-31T21:58:07Z)
• Minimum Bayes Risk Decoding with Neural Metrics of Translation Quality [16.838064121696274]
  This work applies Minimum Bayes Risk decoding to optimize diverse automated metrics of translation quality. Experiments show that the combination of a neural translation model with a neural reference-based metric, BLEURT, results in significant improvement in automatic and human evaluations.
  arXiv  Detail & Related papers  (2021-11-17T20:48:02Z)
• Simulated Multiple Reference Training Improves Low-Resource Machine Translation [22.404646693366054]
  We introduce Simulated Multiple Reference Training (SMRT), a novel MT training method that approximates the full space of possible translations. We demonstrate the effectiveness of SMRT in low-resource settings when translating to English, with improvements of 1.2 to 7.0 BLEU.
  arXiv  Detail & Related papers  (2020-04-30T00:11:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.