On the Inference Calibration of Neural Machine Translation

• URL: http://arxiv.org/abs/2005.00963v1
• Date: Sun, 3 May 2020 02:03:56 GMT
• Title: On the Inference Calibration of Neural Machine Translation
• Authors: Shuo Wang, Zhaopeng Tu, Shuming Shi, Yang Liu
• Abstract summary: We study the correlation between calibration and translation performance and linguistic properties of miscalibration. We propose a new graduated label smoothing method that can improve both inference calibration and translation performance.
• Score: 54.48932804996506
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Confidence calibration, which aims to make model predictions equal to the true correctness measures, is important for neural machine translation (NMT) because it is able to offer useful indicators of translation errors in the generated output. While prior studies have shown that NMT models trained with label smoothing are well-calibrated on the ground-truth training data, we find that miscalibration still remains a severe challenge for NMT during inference due to the discrepancy between training and inference. By carefully designing experiments on three language pairs, our work provides in-depth analyses of the correlation between calibration and translation performance as well as linguistic properties of miscalibration and reports a number of interesting findings that might help humans better analyze, understand and improve NMT models. Based on these observations, we further propose a new graduated label smoothing method that can improve both inference calibration and translation performance.

Related papers

• An Empirical study of Unsupervised Neural Machine Translation: analyzing NMT output, model's behavior and sentences' contribution [5.691028372215281]
  Unsupervised Neural Machine Translation (UNMT) focuses on improving NMT results under the assumption there is no human translated parallel data. We focus on three very diverse languages, French, Gujarati, and Kazakh, and train bilingual NMT models, to and from English, with various levels of supervision.
  arXiv  Detail & Related papers  (2023-12-19T20:35:08Z)
• A Study on the Calibration of In-context Learning [27.533223818505682]
  We study in-context learning (ICL), a prevalent method for adapting static language models through tailored prompts. We observe that, with an increasing number of ICL examples, models initially exhibit increased miscalibration before achieving better calibration. We explore recalibration techniques and find that a scaling-binning calibrator can reduce calibration errors consistently.
  arXiv  Detail & Related papers  (2023-12-07T03:37:39Z)
• Understanding and Mitigating the Uncertainty in Zero-Shot Translation [92.25357943169601]
  We aim to understand and alleviate the off-target issues from the perspective of uncertainty in zero-shot translation. We propose two lightweight and complementary approaches to denoise the training data for model training. Our approaches significantly improve the performance of zero-shot translation over strong MNMT baselines.
  arXiv  Detail & Related papers  (2022-05-20T10:29:46Z)
• SALTED: A Framework for SAlient Long-Tail Translation Error Detection [17.914521288548844]
  We introduce SALTED, a specifications-based framework for behavioral testing of machine translation models. At the core of our approach is the development of high-precision detectors that flag errors between a source sentence and a system output. We demonstrate that such detectors could be used not just to identify salient long-tail errors in MT systems, but also for higher-recall filtering of the training data.
  arXiv  Detail & Related papers  (2022-05-20T06:45:07Z)
• On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency [47.90235939359225]
  We propose a novel mixup strategy for pre-trained language models that improves model calibration further. Our method achieves the lowest expected calibration error compared to strong baselines on both in-domain and out-of-domain test samples.
  arXiv  Detail & Related papers  (2022-03-14T23:45:08Z)
• Non-Parametric Online Learning from Human Feedback for Neural Machine Translation [54.96594148572804]
  We study the problem of online learning with human feedback in the human-in-the-loop machine translation. Previous methods require online model updating or additional translation memory networks to achieve high-quality performance. We propose a novel non-parametric online learning method without changing the model structure.
  arXiv  Detail & Related papers  (2021-09-23T04:26:15Z)
• Improving Multilingual Translation by Representation and Gradient Regularization [82.42760103045083]
  We propose a joint approach to regularize NMT models at both representation-level and gradient-level. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance.
  arXiv  Detail & Related papers  (2021-09-10T10:52:21Z)
• Alternated Training with Synthetic and Authentic Data for Neural Machine Translation [49.35605028467887]
  We propose alternated training with synthetic and authentic data for neural machine translation (NMT) Compared with previous work, we introduce authentic data as guidance to prevent the training of NMT models from being disturbed by noisy synthetic data. Experiments on Chinese-English and German-English translation tasks show that our approach improves the performance over several strong baselines.
  arXiv  Detail & Related papers  (2021-06-16T07:13:16Z)
• Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation [14.645468999921961]
  We analyze the impact of different types of fine-grained semantic divergences on Transformer models. We introduce a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences.
  arXiv  Detail & Related papers  (2021-05-31T16:15:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.