Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation

• URL: http://arxiv.org/abs/2212.09631v2
• Date: Fri, 19 May 2023 16:36:12 GMT
• Title: Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation
• Authors: Nuno M. Guerreiro, Pierre Colombo, Pablo Piantanida, Andr\'e F. T. Martins
• Abstract summary: Neural machine translation (NMT) has become the de-facto standard in real-world machine translation applications. NMT models can unpredictably produce severely pathological translations, known as hallucinations, that seriously undermine user trust. We propose a fully unsupervised, plug-in detector that can be used with any attention-based NMT model.
• Score: 34.8089664250053
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural machine translation (NMT) has become the de-facto standard in real-world machine translation applications. However, NMT models can unpredictably produce severely pathological translations, known as hallucinations, that seriously undermine user trust. It becomes thus crucial to implement effective preventive strategies to guarantee their proper functioning. In this paper, we address the problem of hallucination detection in NMT by following a simple intuition: as hallucinations are detached from the source content, they exhibit encoder-decoder attention patterns that are statistically different from those of good quality translations. We frame this problem with an optimal transport formulation and propose a fully unsupervised, plug-in detector that can be used with any attention-based NMT model. Experimental results show that our detector not only outperforms all previous model-based detectors, but is also competitive with detectors that employ large models trained on millions of samples.

Related papers

• OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection [36.59354124910338]
  Ottawa is a word aligner specifically designed to enhance the detection of hallucinations and omissions in Machine Translation systems. Our approach yields competitive results compared to state-of-the-art methods across 18 language pairs on the HalOmi benchmark.
  arXiv  Detail & Related papers  (2024-06-04T03:00:55Z)
• TransFool: An Adversarial Attack against Neural Machine Translation Models [49.50163349643615]
  We investigate the vulnerability of Neural Machine Translation (NMT) models to adversarial attacks and propose a new attack algorithm called TransFool. We generate fluent adversarial examples in the source language that maintain a high level of semantic similarity with the clean samples. Based on automatic and human evaluations, TransFool leads to improvement in terms of success rate, semantic similarity, and fluency compared to the existing attacks.
  arXiv  Detail & Related papers  (2023-02-02T08:35:34Z)
• Reducing Hallucinations in Neural Machine Translation with Feature Attribution [54.46113444757899]
  We present a case study focusing on model understanding and regularisation to reduce hallucinations in NMT. We first use feature attribution methods to study the behaviour of an NMT model that produces hallucinations. We then leverage these methods to propose a novel loss function that substantially helps reduce hallucinations and does not require retraining the model from scratch.
  arXiv  Detail & Related papers  (2022-11-17T20:33:56Z)
• Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation [17.102338932907294]
  We set foundations for the study of NMT hallucinations. We propose DeHallucinator, a simple method for alleviating hallucinations at test time.
  arXiv  Detail & Related papers  (2022-08-10T12:44:13Z)
• SALTED: A Framework for SAlient Long-Tail Translation Error Detection [17.914521288548844]
  We introduce SALTED, a specifications-based framework for behavioral testing of machine translation models. At the core of our approach is the development of high-precision detectors that flag errors between a source sentence and a system output. We demonstrate that such detectors could be used not just to identify salient long-tail errors in MT systems, but also for higher-recall filtering of the training data.
  arXiv  Detail & Related papers  (2022-05-20T06:45:07Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)
• Detecting Hallucinated Content in Conditional Neural Sequence Generation [165.68948078624499]
  We propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data.
  arXiv  Detail & Related papers  (2020-11-05T00:18:53Z)
• Cross-lingual Supervision Improves Unsupervised Neural Machine Translation [97.84871088440102]
  We introduce a multilingual unsupervised NMT framework to leverage weakly supervised signals from high-resource language pairs to zero-resource translation directions. Method significantly improves the translation quality by more than 3 BLEU score on six benchmark unsupervised translation directions.
  arXiv  Detail & Related papers  (2020-04-07T05:46:49Z)
• Robust Unsupervised Neural Machine Translation with Adversarial Denoising Training [66.39561682517741]
  Unsupervised neural machine translation (UNMT) has attracted great interest in the machine translation community. The main advantage of the UNMT lies in its easy collection of required large training text sentences. In this paper, we first time explicitly take the noisy data into consideration to improve the robustness of the UNMT based systems.
  arXiv  Detail & Related papers  (2020-02-28T05:17:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.