Quality-Aware Decoding: Unifying Quality Estimation and Decoding

• URL: http://arxiv.org/abs/2502.08561v2
• Date: Sun, 16 Feb 2025 13:43:39 GMT
• Title: Quality-Aware Decoding: Unifying Quality Estimation and Decoding
• Authors: Sai Koneru, Matthias Huck, Miriam Exel, Jan Niehues,
• Abstract summary: We present a novel token-level QE model capable of reliably scoring partial translations. We then present a decoding strategy that integrates the QE model for Quality-Aware decoding. Our approach provides significant benefits in document translation tasks.
• Score: 12.843274390224853
• License:
• Abstract: Quality Estimation (QE) models for Neural Machine Translation (NMT) predict the quality of the hypothesis without having access to the reference. An emerging research direction in NMT involves the use of QE models, which have demonstrated high correlations with human judgment and can enhance translations through Quality-Aware Decoding. Although several approaches have been proposed based on sampling multiple candidate translations and picking the best candidate, none have integrated these models directly into the decoding process. In this paper, we address this by proposing a novel token-level QE model capable of reliably scoring partial translations. We build a uni-directional QE model for this, as decoder models are inherently trained and efficient on partial sequences. We then present a decoding strategy that integrates the QE model for Quality-Aware decoding and demonstrate that the translation quality improves when compared to the N-best list re-ranking with state-of-the-art QE models (up to $1.39$ XCOMET-XXL $\uparrow$). Finally, we show that our approach provides significant benefits in document translation tasks, where the quality of N-best lists is typically suboptimal. Code can be found at https://ai4lt.iar.kit.edu/english/projects\_kontextmt.php

Related papers

• When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages [9.138590152838754]
  Segment-level quality estimation (QE) is a challenging cross-lingual language understanding task. We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios. Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models.
  arXiv  Detail & Related papers  (2025-01-08T12:54:05Z)
• A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning [49.62044186504516]
  In document-level neural machine translation (DocNMT), multi-encoder approaches are common in encoding context and source sentences. Recent studies have shown that the context encoder generates noise and makes the model robust to the choice of context. This paper further investigates this observation by explicitly modelling context encoding through multi-task learning (MTL) to make the model sensitive to the choice of context.
  arXiv  Detail & Related papers  (2024-07-03T12:50:49Z)
• Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean [7.843029855730508]
  We develop a 1200-sentence MQM evaluation benchmark for the language pair English-Korean. We find that reference-free setup outperforms its counterpart in the style dimension. Overall, RemBERT emerges as the most promising model.
  arXiv  Detail & Related papers  (2024-03-19T12:02:38Z)
• Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation [0.6998085564793366]
  This work introduces QE-fusion, a method that synthesizes translations using a quality estimation metric (QE) We demonstrate that our approach generates novel translations in over half of the cases. We empirically establish that QE-fusion scales linearly with the number of candidates in the pool.
  arXiv  Detail & Related papers  (2024-01-12T16:52:41Z)
• Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Model [77.19693792957614]
  We propose to make neural machine translation (NMT) models quality-aware by training them to estimate the quality of their own output. We obtain quality gains similar or even superior to quality reranking approaches, but with the efficiency of single pass decoding.
  arXiv  Detail & Related papers  (2023-10-10T15:33:51Z)
• On Search Strategies for Document-Level Neural Machine Translation [51.359400776242786]
  Document-level neural machine translation (NMT) models produce a more consistent output across a document. In this work, we aim to answer the question how to best utilize a context-aware translation model in decoding.
  arXiv  Detail & Related papers  (2023-06-08T11:30:43Z)
• Generative Language Models for Paragraph-Level Question Generation [79.31199020420827]
  Powerful generative models have led to recent progress in question generation (QG) It is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. We introduce QG-Bench, a benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting.
  arXiv  Detail & Related papers  (2022-10-08T10:24:39Z)
• Quality-Aware Decoding for Neural Machine Translation [64.24934199944875]
  We propose quality-aware decoding for neural machine translation (NMT) We leverage recent breakthroughs in reference-free and reference-based MT evaluation through various inference methods. We find that quality-aware decoding consistently outperforms MAP-based decoding according both to state-of-the-art automatic metrics and to human assessments.
  arXiv  Detail & Related papers  (2022-05-02T15:26:28Z)
• Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation [88.78138830698173]
  We focus on sequence-level knowledge distillation (SeqKD) from external text-based NMT models. We train a bilingual E2E-ST model to predict paraphrased transcriptions as an auxiliary task with a single decoder.
  arXiv  Detail & Related papers  (2021-04-13T19:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.