QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation

• URL: http://arxiv.org/abs/2406.00049v2
• Date: Tue, 15 Oct 2024 18:30:29 GMT
• Title: QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation
• Authors: Gonçalo R. A. Faria, Sweta Agrawal, António Farinhas, Ricardo Rei, José G. C. de Souza, André F. T. Martins,
• Abstract summary: We propose a simple and effective way to avoid over-reliance on noisy quality estimates by using them as the energy function of a Gibbs distribution. Instead of looking for a mode in the distribution, we generate multiple samples from high-density areas through the Metropolis-Hastings algorithm.
• Score: 25.165239478219267
• License:
• Abstract: An important challenge in machine translation (MT) is to generate high-quality and diverse translations. Prior work has shown that the estimated likelihood from the MT model correlates poorly with translation quality. In contrast, quality evaluation metrics (such as COMET or BLEURT) exhibit high correlations with human judgments, which has motivated their use as rerankers (such as quality-aware and minimum Bayes risk decoding). However, relying on a single translation with high estimated quality increases the chances of "gaming the metric''. In this paper, we address the problem of sampling a set of high-quality and diverse translations. We provide a simple and effective way to avoid over-reliance on noisy quality estimates by using them as the energy function of a Gibbs distribution. Instead of looking for a mode in the distribution, we generate multiple samples from high-density areas through the Metropolis-Hastings algorithm, a simple Markov chain Monte Carlo approach. The results show that our proposed method leads to high-quality and diverse outputs across multiple language pairs (English$\leftrightarrow${German, Russian}) with two strong decoder-only LLMs (Alma-7b, Tower-7b).

Related papers

• MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators [53.91199933655421]
  Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment. We introduce a universal and training-free framework, $textbfMQM-APE, to enhance the quality of error annotations predicted by LLM evaluators.
  arXiv  Detail & Related papers  (2024-09-22T06:43:40Z)
• A Data Selection Approach for Enhancing Low Resource Machine Translation Using Cross-Lingual Sentence Representations [0.4499833362998489]
  This study focuses on the case of English-Marathi language pairs, where existing datasets are notably noisy. To mitigate the impact of data quality issues, we propose a data filtering approach based on cross-lingual sentence representations. Results demonstrate a significant improvement in translation quality over the baseline post-filtering with IndicSBERT.
  arXiv  Detail & Related papers  (2024-09-04T13:49:45Z)
• Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation [0.6998085564793366]
  This work introduces QE-fusion, a method that synthesizes translations using a quality estimation metric (QE) We demonstrate that our approach generates novel translations in over half of the cases. We empirically establish that QE-fusion scales linearly with the number of candidates in the pool.
  arXiv  Detail & Related papers  (2024-01-12T16:52:41Z)
• Exploring Human-Like Translation Strategy with Large Language Models [93.49333173279508]
  Large language models (LLMs) have demonstrated impressive capabilities in general scenarios. This work proposes the MAPS framework, which stands for Multi-Aspect Prompting and Selection. We employ a selection mechanism based on quality estimation to filter out noisy and unhelpful knowledge.
  arXiv  Detail & Related papers  (2023-05-06T19:03:12Z)
• Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation [72.6667341525552]
  We present a new MMT approach based on a strong text-only MT model, which uses neural adapters and a novel guided self-attention mechanism. We also introduce CoMMuTE, a Contrastive Multimodal Translation Evaluation set of ambiguous sentences and their possible translations. Our approach obtains competitive results compared to strong text-only models on standard English-to-French, English-to-German and English-to-Czech benchmarks.
  arXiv  Detail & Related papers  (2022-12-20T10:18:18Z)
• Mismatching-Aware Unsupervised Translation Quality Estimation For Low-Resource Languages [6.049660810617423]
  XLMRScore is a cross-lingual counterpart of BERTScore computed via the XLM-RoBERTa (XLMR) model. We evaluate the proposed method on four low-resource language pairs of the WMT21 QE shared task.
  arXiv  Detail & Related papers  (2022-07-31T16:23:23Z)
• Quality-Aware Decoding for Neural Machine Translation [64.24934199944875]
  We propose quality-aware decoding for neural machine translation (NMT) We leverage recent breakthroughs in reference-free and reference-based MT evaluation through various inference methods. We find that quality-aware decoding consistently outperforms MAP-based decoding according both to state-of-the-art automatic metrics and to human assessments.
  arXiv  Detail & Related papers  (2022-05-02T15:26:28Z)
• HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation [0.0]
  In this work, we introduce HOPE, a task-oriented and human-centric evaluation framework for machine translation output. It contains only a limited number of commonly occurring error types, and use a scoring model with geometric progression of error penalty points (EPPs) reflecting error severity level to each translation unit. The approach has several key advantages, such as ability to measure and compare less than perfect MT output from different systems, ability to indicate human perception of quality, immediate estimation of the labor effort required to bring MT output to premium quality, low-cost and faster application, as well as higher IRR.
  arXiv  Detail & Related papers  (2021-12-27T18:47:43Z)
• Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
  This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
  arXiv  Detail & Related papers  (2021-11-15T12:09:08Z)
• Modelling Latent Translations for Cross-Lingual Transfer [47.61502999819699]
  We propose a new technique that integrates both steps of the traditional pipeline (translation and classification) into a single model. We evaluate our novel latent translation-based model on a series of multilingual NLU tasks. We report gains for both zero-shot and few-shot learning setups, up to 2.7 accuracy points on average.
  arXiv  Detail & Related papers  (2021-07-23T17:11:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.