End-to-End Training for Back-Translation with Categorical Reparameterization Trick

• URL: http://arxiv.org/abs/2202.08465v4
• Date: Sat, 29 Jun 2024 08:00:04 GMT
• Title: End-to-End Training for Back-Translation with Categorical Reparameterization Trick
• Authors: DongNyeong Heo, Heeyoul Choi,
• Abstract summary: Back-translation is an effective semi-supervised learning framework in neural machine translation (NMT) A pre-trained NMT model translates monolingual sentences and makes synthetic bilingual sentence pairs for the training of the other NMT model. The discrete property of translated sentences prevents information gradient from flowing between the two NMT models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Back-translation (BT) is an effective semi-supervised learning framework in neural machine translation (NMT). A pre-trained NMT model translates monolingual sentences and makes synthetic bilingual sentence pairs for the training of the other NMT model, and vice versa. Understanding the two NMT models as inference and generation models, respectively, the training method of variational auto-encoder (VAE) was applied in previous works, which is a mainstream framework of generative models. However, the discrete property of translated sentences prevents gradient information from flowing between the two NMT models. In this paper, we propose the categorical reparameterization trick (CRT) that makes NMT models generate differentiable sentences so that the VAE's training framework can work in an end-to-end fashion. Our BT experiment conducted on a WMT benchmark dataset demonstrates the superiority of our proposed CRT compared to the Gumbel-softmax trick, which is a popular reparameterization method for categorical variable. Moreover, our experiments conducted on multiple WMT benchmark datasets demonstrate that our proposed end-to-end training framework is effective in terms of BLEU scores not only compared to its counterpart baseline which is not trained in an end-to-end fashion, but also compared to other previous BT works. The code is available at the web.

Related papers

• Choose the Final Translation from NMT and LLM hypotheses Using MBR Decoding: HW-TSC's Submission to the WMT24 General MT Shared Task [9.819139035652137]
  This paper presents the submission of Huawei Translate Services Center (HW-TSC) to the WMT24 general machine translation (MT) shared task. We use training strategies such as regularized dropout, bidirectional training, data diversification, forward translation, back translation, alternated training, curriculum learning, and transductive ensemble learning to train the neural machine translation (NMT) model.
  arXiv  Detail & Related papers  (2024-09-23T08:25:37Z)
• Better Datastore, Better Translation: Generating Datastores from Pre-Trained Models for Nearest Neural Machine Translation [48.58899349349702]
  Nearest Neighbor Machine Translation (kNNMT) is a simple and effective method of augmenting neural machine translation (NMT) with a token-level nearest neighbor retrieval mechanism. In this paper, we propose PRED, a framework that leverages Pre-trained models for Datastores in kNN-MT.
  arXiv  Detail & Related papers  (2022-12-17T08:34:20Z)
• Data Selection Curriculum for Neural Machine Translation [31.55953464971441]
  We introduce a two-stage curriculum training framework for NMT models. We fine-tune a base NMT model on subsets of data, selected by both deterministic scoring using pre-trained methods and online scoring. We have shown that our curriculum strategies consistently demonstrate better quality (up to +2.2 BLEU improvement) and faster convergence.
  arXiv  Detail & Related papers  (2022-03-25T19:08:30Z)
• Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT [64.1841519527504]
  neural machine translation uses a single neural network to model the entire translation process. Despite neural machine translation being de-facto standard, it is still not clear how NMT models acquire different competences over the course of training.
  arXiv  Detail & Related papers  (2021-09-03T09:38:50Z)
• The USYD-JD Speech Translation System for IWSLT 2021 [85.64797317290349]
  This paper describes the University of Sydney& JD's joint submission of the IWSLT 2021 low resource speech translation task. We trained our models with the officially provided ASR and MT datasets. To achieve better translation performance, we explored the most recent effective strategies, including back translation, knowledge distillation, multi-feature reranking and transductive finetuning.
  arXiv  Detail & Related papers  (2021-07-24T09:53:34Z)
• Alternated Training with Synthetic and Authentic Data for Neural Machine Translation [49.35605028467887]
  We propose alternated training with synthetic and authentic data for neural machine translation (NMT) Compared with previous work, we introduce authentic data as guidance to prevent the training of NMT models from being disturbed by noisy synthetic data. Experiments on Chinese-English and German-English translation tasks show that our approach improves the performance over several strong baselines.
  arXiv  Detail & Related papers  (2021-06-16T07:13:16Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)
• Self-supervised and Supervised Joint Training for Resource-rich Machine Translation [30.502625878505732]
  Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT) We propose a joint training approach, $F$-XEnDec, to combine self-supervised and supervised learning to optimize NMT models.
  arXiv  Detail & Related papers  (2021-06-08T02:35:40Z)
• Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation [59.191079800436114]
  Document-level machine translation incorporates inter-sentential dependencies into the translation of a source sentence. We propose a new framework to model cross-sentence dependencies by training neural machine translation (NMT) to predict both the target translation and surrounding sentences of a source sentence.
  arXiv  Detail & Related papers  (2020-03-30T03:38:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.