Testing Machine Translation via Referential Transparency
- URL: http://arxiv.org/abs/2004.10361v2
- Date: Sun, 28 Feb 2021 12:56:49 GMT
- Title: Testing Machine Translation via Referential Transparency
- Authors: Pinjia He, Clara Meister, Zhendong Su
- Abstract summary: We introduce referentially transparent inputs (RTIs), a simple, widely applicable methodology for validating machine translation software.
Our practical implementation, Purity, detects when this property is broken by a translation.
To evaluate RTI, we use Purity to test Google Translate and Bing Microsoft Translator with 200 unlabeled sentences.
- Score: 28.931196266344926
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Machine translation software has seen rapid progress in recent years due to
the advancement of deep neural networks. People routinely use machine
translation software in their daily lives, such as ordering food in a foreign
restaurant, receiving medical diagnosis and treatment from foreign doctors, and
reading international political news online. However, due to the complexity and
intractability of the underlying neural networks, modern machine translation
software is still far from robust and can produce poor or incorrect
translations; this can lead to misunderstanding, financial loss, threats to
personal safety and health, and political conflicts. To address this problem,
we introduce referentially transparent inputs (RTIs), a simple, widely
applicable methodology for validating machine translation software. A
referentially transparent input is a piece of text that should have similar
translations when used in different contexts. Our practical implementation,
Purity, detects when this property is broken by a translation. To evaluate RTI,
we use Purity to test Google Translate and Bing Microsoft Translator with 200
unlabeled sentences, which detected 123 and 142 erroneous translations with
high precision (79.3% and 78.3%). The translation errors are diverse, including
examples of under-translation, over-translation, word/phrase mistranslation,
incorrect modification, and unclear logic.
Related papers
- Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective [72.83966378613238]
Under-translation and over-translation remain two challenging problems in state-of-the-art Neural Machine Translation (NMT) systems.
We conduct an in-depth analysis on the underlying cause of under-translation in NMT, providing an explanation from the perspective of decoding objective.
We propose employing the confidence of predicting End Of Sentence (EOS) as a detector for under-translation, and strengthening the confidence-based penalty to penalize candidates with a high risk of under-translation.
arXiv Detail & Related papers (2024-05-29T09:25:49Z) - Machine Translation Models are Zero-Shot Detectors of Translation Direction [46.41883195574249]
Detecting the translation direction of parallel text has applications for machine translation training and evaluation, but also has forensic applications such as resolving plagiarism or forgery allegations.
In this work, we explore an unsupervised approach to translation direction detection based on the simple hypothesis that $p(texttranslation|textoriginal)>p(textoriginal|texttranslation)$, motivated by the well-known simplification effect in translationese or machine-translationese.
arXiv Detail & Related papers (2024-01-12T18:59:02Z) - Machine Translation Testing via Syntactic Tree Pruning [19.023809217746955]
erroneous translations may result in severe consequences, such as financial losses.
It is challenging to test machine translation systems because of the complexity and intractability of the underlying neural models.
arXiv Detail & Related papers (2024-01-01T13:28:46Z) - Crossing the Threshold: Idiomatic Machine Translation through Retrieval
Augmentation and Loss Weighting [66.02718577386426]
We provide a simple characterization of idiomatic translation and related issues.
We conduct a synthetic experiment revealing a tipping point at which transformer-based machine translation models correctly default to idiomatic translations.
To improve translation of natural idioms, we introduce two straightforward yet effective techniques.
arXiv Detail & Related papers (2023-10-10T23:47:25Z) - Mitigating Hallucinations and Off-target Machine Translation with
Source-Contrastive and Language-Contrastive Decoding [53.84948040596055]
We introduce two related methods to mitigate failure cases with a modified decoding objective.
Experiments on the massively multilingual models M2M-100 (418M) and SMaLL-100 show that these methods suppress hallucinations and off-target translations.
arXiv Detail & Related papers (2023-09-13T17:15:27Z) - The Best of Both Worlds: Combining Human and Machine Translations for
Multilingual Semantic Parsing with Active Learning [50.320178219081484]
We propose an active learning approach that exploits the strengths of both human and machine translations.
An ideal utterance selection can significantly reduce the error and bias in the translated data.
arXiv Detail & Related papers (2023-05-22T05:57:47Z) - Translate your gibberish: black-box adversarial attack on machine
translation systems [0.0]
We present a simple approach to fool state-of-the-art machine translation tools in the task of translation from Russian to English and vice versa.
We show that many online translation tools, such as Google, DeepL, and Yandex, may both produce wrong or offensive translations for nonsensical adversarial input queries.
This vulnerability may interfere with understanding a new language and simply worsen the user's experience while using machine translation systems.
arXiv Detail & Related papers (2023-03-20T09:52:52Z) - Easy Guided Decoding in Providing Suggestions for Interactive Machine
Translation [14.615314828955288]
We propose a novel constrained decoding algorithm, namely Prefix Suffix Guided Decoding (PSGD)
PSGD improves translation quality by an average of $10.87$ BLEU and $8.62$ BLEU on the WeTS and the WMT 2022 Translation Suggestion datasets.
arXiv Detail & Related papers (2022-11-14T03:40:02Z) - Leveraging Automated Unit Tests for Unsupervised Code Translation [34.84910520660154]
We propose to leverage an automated unit-testing system to filter out invalid translations.
We find that fine-tuning an unsupervised model with this filtered data set significantly reduces the noise in the translations so-generated.
In particular, for Java $to$ Python and Python $to$ C++ we outperform the best previous methods by more than 16% and 24% respectively.
arXiv Detail & Related papers (2021-10-13T15:08:43Z) - It's Easier to Translate out of English than into it: Measuring Neural
Translation Difficulty by Cross-Mutual Information [90.35685796083563]
Cross-mutual information (XMI) is an asymmetric information-theoretic metric of machine translation difficulty.
XMI exploits the probabilistic nature of most neural machine translation models.
We present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems.
arXiv Detail & Related papers (2020-05-05T17:38:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.