Sometimes We Want Translationese
- URL: http://arxiv.org/abs/2104.07623v1
- Date: Thu, 15 Apr 2021 17:39:47 GMT
- Title: Sometimes We Want Translationese
- Authors: Prasanna Parthasarathi, Koustuv Sinha, Joelle Pineau and Adina
Williams
- Abstract summary: In some applications, faithfulness to the original (input) text is important to preserve.
We propose a simple, novel way to quantify whether an NMT system exhibits robustness and faithfulness.
- Score: 48.45003475966808
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Rapid progress in Neural Machine Translation (NMT) systems over the last few
years has been driven primarily towards improving translation quality, and as a
secondary focus, improved robustness to input perturbations (e.g. spelling and
grammatical mistakes). While performance and robustness are important
objectives, by over-focusing on these, we risk overlooking other important
properties. In this paper, we draw attention to the fact that for some
applications, faithfulness to the original (input) text is important to
preserve, even if it means introducing unusual language patterns in the
(output) translation. We propose a simple, novel way to quantify whether an NMT
system exhibits robustness and faithfulness, focusing on the case of word-order
perturbations. We explore a suite of functions to perturb the word order of
source sentences without deleting or injecting tokens, and measure the effects
on the target side in terms of both robustness and faithfulness. Across several
experimental conditions, we observe a strong tendency towards robustness rather
than faithfulness. These results allow us to better understand the trade-off
between faithfulness and robustness in NMT, and opens up the possibility of
developing systems where users have more autonomy and control in selecting
which property is best suited for their use case.
Related papers
- Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective [72.83966378613238]
Under-translation and over-translation remain two challenging problems in state-of-the-art Neural Machine Translation (NMT) systems.
We conduct an in-depth analysis on the underlying cause of under-translation in NMT, providing an explanation from the perspective of decoding objective.
We propose employing the confidence of predicting End Of Sentence (EOS) as a detector for under-translation, and strengthening the confidence-based penalty to penalize candidates with a high risk of under-translation.
arXiv Detail & Related papers (2024-05-29T09:25:49Z) - Towards preserving word order importance through Forced Invalidation [80.33036864442182]
We show that pre-trained language models are insensitive to word order.
We propose Forced Invalidation to help preserve the importance of word order.
Our experiments demonstrate that Forced Invalidation significantly improves the sensitivity of the models to word order.
arXiv Detail & Related papers (2023-04-11T13:42:10Z) - Competency-Aware Neural Machine Translation: Can Machine Translation
Know its Own Translation Quality? [61.866103154161884]
Neural machine translation (NMT) is often criticized for failures that happen without awareness.
We propose a novel competency-aware NMT by extending conventional NMT with a self-estimator.
We show that the proposed method delivers outstanding performance on quality estimation.
arXiv Detail & Related papers (2022-11-25T02:39:41Z) - Understanding and Mitigating the Uncertainty in Zero-Shot Translation [66.49354195238789]
We aim to understand and alleviate the off-target issues from the perspective of uncertainty in zero-shot translation.
By carefully examining the translation output and model confidence, we identify two uncertainties that are responsible for the off-target issues.
We propose two light-weight and complementary approaches to denoise the training data for model training, and mask out the vocabulary of the off-target languages in inference.
arXiv Detail & Related papers (2022-05-20T10:29:46Z) - PreQuEL: Quality Estimation of Machine Translation Outputs in Advance [32.922128367314194]
A PreQuEL system predicts how well a given sentence will be translated, without recourse to the actual translation.
We develop a baseline model for the task and analyze its performance.
We show that this augmentation method can improve the performance of the Quality-Estimation task as well.
arXiv Detail & Related papers (2022-05-18T18:55:05Z) - DEEP: DEnoising Entity Pre-training for Neural Machine Translation [123.6686940355937]
It has been shown that machine translation models usually generate poor translations for named entities that are infrequent in the training corpus.
We propose DEEP, a DEnoising Entity Pre-training method that leverages large amounts of monolingual data and a knowledge base to improve named entity translation accuracy within sentences.
arXiv Detail & Related papers (2021-11-14T17:28:09Z) - PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on
User-Generated Contents [40.25277134147149]
We present a new dataset, PheMT, for evaluating the robustness of MT systems against specific linguistic phenomena in Japanese-English translation.
Our experiments with the created dataset revealed that not only our in-house models but even widely used off-the-shelf systems are greatly disturbed by the presence of certain phenomena.
arXiv Detail & Related papers (2020-11-04T04:44:47Z) - Uncertainty-Aware Semantic Augmentation for Neural Machine Translation [37.555675157198145]
We propose uncertainty-aware semantic augmentation, which explicitly captures the universal semantic information among multiple semantically-equivalent source sentences.
Our approach significantly outperforms the strong baselines and the existing methods.
arXiv Detail & Related papers (2020-10-09T07:48:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.