Quality-Aware Decoding for Neural Machine Translation

• URL: http://arxiv.org/abs/2205.00978v1
• Date: Mon, 2 May 2022 15:26:28 GMT
• Title: Quality-Aware Decoding for Neural Machine Translation
• Authors: Patrick Fernandes, Ant\'onio Farinhas, Ricardo Rei, Jos\'e G. C. de Souza, Perez Ogayo, Graham Neubig, Andr\'e F. T. Martins
• Abstract summary: We propose quality-aware decoding for neural machine translation (NMT) We leverage recent breakthroughs in reference-free and reference-based MT evaluation through various inference methods. We find that quality-aware decoding consistently outperforms MAP-based decoding according both to state-of-the-art automatic metrics and to human assessments.
• Score: 64.24934199944875
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the progress in machine translation quality estimation and evaluation in the last years, decoding in neural machine translation (NMT) is mostly oblivious to this and centers around finding the most probable translation according to the model (MAP decoding), approximated with beam search. In this paper, we bring together these two lines of research and propose quality-aware decoding for NMT, by leveraging recent breakthroughs in reference-free and reference-based MT evaluation through various inference methods like $N$-best reranking and minimum Bayes risk decoding. We perform an extensive comparison of various possible candidate generation and ranking methods across four datasets and two model classes and find that quality-aware decoding consistently outperforms MAP-based decoding according both to state-of-the-art automatic metrics (COMET and BLEURT) and to human assessments. Our code is available at https://github.com/deep-spin/qaware-decode.

Related papers

• Linear-time Minimum Bayes Risk Decoding with Reference Aggregation [52.1701152610258]
  Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations. It requires the pairwise calculation of a utility metric, which has quadratic complexity. We propose to approximate pairwise metric scores with scores calculated against aggregated reference representations.
  arXiv  Detail & Related papers  (2024-02-06T18:59:30Z)
• Exploring Automatic Evaluation Methods based on a Decoder-based LLM for Text Generation [16.78350863261211]
  This paper compares various methods, including tuning with encoder-based models and large language models under equal conditions. Experimental results show that compared to the tuned encoder-based models, the tuned decoder-based models perform poorly. It is also revealed that in-context learning of very large decoder-based models such as ChatGPT makes it difficult to identify fine-grained semantic differences.
  arXiv  Detail & Related papers  (2023-10-17T06:53:00Z)
• Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Model [77.19693792957614]
  We propose to make neural machine translation (NMT) models quality-aware by training them to estimate the quality of their own output. We obtain quality gains similar or even superior to quality reranking approaches, but with the efficiency of single pass decoding.
  arXiv  Detail & Related papers  (2023-10-10T15:33:51Z)
• On Search Strategies for Document-Level Neural Machine Translation [51.359400776242786]
  Document-level neural machine translation (NMT) models produce a more consistent output across a document. In this work, we aim to answer the question how to best utilize a context-aware translation model in decoding.
  arXiv  Detail & Related papers  (2023-06-08T11:30:43Z)
• Neural Machine Translation for Code Generation [0.7607163273993514]
  In NMT for code generation, the task is to generate source code that satisfies constraints expressed in the input. In this paper we survey the NMT for code generation literature, cataloging the variety of methods that have been explored. We discuss the limitations of existing methods and future research directions.
  arXiv  Detail & Related papers  (2023-05-22T21:43:12Z)
• Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? [61.866103154161884]
  Neural machine translation (NMT) is often criticized for failures that happen without awareness. We propose a novel competency-aware NMT by extending conventional NMT with a self-estimator. We show that the proposed method delivers outstanding performance on quality estimation.
  arXiv  Detail & Related papers  (2022-11-25T02:39:41Z)
• FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation [64.9546787488337]
  We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese.
  arXiv  Detail & Related papers  (2022-10-01T05:02:04Z)
• Rethinking the Evaluation of Neural Machine Translation [25.036685025571927]
  We propose a novel evaluation protocol, which avoids the effect of search errors and provides a system-level evaluation in the perspective of model ranking. Our method is based on our newly proposed exact top-$k$ decoding instead of beam search.
  arXiv  Detail & Related papers  (2021-06-29T09:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.