On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation

• URL: http://arxiv.org/abs/2110.01811v1
• Date: Tue, 5 Oct 2021 04:01:36 GMT
• Title: On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation
• Authors: Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao, Shuming Shi, Zhaopeng Tu
• Abstract summary: Pre-training (PT) and back-translation (BT) are two simple and powerful methods to utilize monolingual data. This paper takes the first step to investigate the complementarity between PT and BT. We establish state-of-the-art performances on the WMT16 English-Romanian and English-Russian benchmarks.
• Score: 63.914940899327966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-training (PT) and back-translation (BT) are two simple and powerful methods to utilize monolingual data for improving the model performance of neural machine translation (NMT). This paper takes the first step to investigate the complementarity between PT and BT. We introduce two probing tasks for PT and BT respectively and find that PT mainly contributes to the encoder module while BT brings more benefits to the decoder. Experimental results show that PT and BT are nicely complementary to each other, establishing state-of-the-art performances on the WMT16 English-Romanian and English-Russian benchmarks. Through extensive analyses on sentence originality and word frequency, we also demonstrate that combining Tagged BT with PT is more helpful to their complementarity, leading to better translation quality. Source code is freely available at https://github.com/SunbowLiu/PTvsBT.

Related papers

• Text Style Transfer Back-Translation [14.608570096595177]
  Back Translation improves translation of inputs that share a similar style. For natural inputs, BT brings only slight improvements and sometimes even adverse effects. We propose Text Style Transfer Back Translation, which uses a style transfer model to modify the source side of BT data.
  arXiv  Detail & Related papers  (2023-06-02T07:33:47Z)
• Revamping Multilingual Agreement Bidirectionally via Switched Back-translation for Multilingual Neural Machine Translation [107.83158521848372]
  multilingual agreement (MA) has shown its importance for multilingual neural machine translation (MNMT) We present textbfBidirectional textbfMultilingual textbfAgreement via textbfSwitched textbfBack-textbftranslation (textbfBMA-SBT) It is a novel and universal multilingual agreement framework for fine-tuning pre-trained MNMT models.
  arXiv  Detail & Related papers  (2022-09-28T09:14:58Z)
• On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation [80.16548523140025]
  Pre-Training (PT) of text representations has been successfully applied to low-resource Neural Machine Translation (NMT) We propose to combine their complementarities with a model fusion algorithm that utilizes optimal transport to align neurons between PT and RI. Experiments on two resource-rich translation benchmarks, WMT'17 English-Chinese (20M) and WMT'19 English-German (36M), show that PT and RI could be nicely complementary to each other.
  arXiv  Detail & Related papers  (2022-09-07T17:23:08Z)
• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation [76.13334392868208]
  Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues. In this work, we explore self-supervised pre-training with unlabeled speech data and data augmentation to tackle this issue.
  arXiv  Detail & Related papers  (2022-04-06T17:59:22Z)
• Unsupervised Bitext Mining and Translation via Self-trained Contextual Embeddings [51.47607125262885]
  We describe an unsupervised method to create pseudo-parallel corpora for machine translation (MT) from unaligned text. We use multilingual BERT to create source and target sentence embeddings for nearest-neighbor search and adapt the model via self-training. We validate our technique by extracting parallel sentence pairs on the BUCC 2017 bitext mining task and observe up to a 24.5 point increase (absolute) in F1 scores over previous unsupervised methods.
  arXiv  Detail & Related papers  (2020-10-15T14:04:03Z)
• SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task [111.91077204077817]
  We participated in four translation directions of three language pairs: English-Chinese, English-Polish, and German-Upper Sorbian. Based on different conditions of language pairs, we have experimented with diverse neural machine translation (NMT) techniques. In our submissions, the primary systems won the first place on English to Chinese, Polish to English, and German to Upper Sorbian translation directions.
  arXiv  Detail & Related papers  (2020-10-11T00:40:05Z)
• Evaluating Low-Resource Machine Translation between Chinese and Vietnamese with Back-Translation [32.25731930652532]
  Back translation (BT) has been widely used and become one of standard techniques for data augmentation in Neural Machine Translation (NMT) We evaluate and compare the effects of different sizes of synthetic data on both NMT and Statistical Machine Translation (SMT) models for Chinese to Vietnamese and Vietnamese to Chinese, with character-based and word-based settings.
  arXiv  Detail & Related papers  (2020-03-04T17:10:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.