Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation

• URL: http://arxiv.org/abs/2010.12868v2
• Date: Mon, 17 May 2021 07:24:55 GMT
• Title: Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation
• Authors: Yongchang Hao, Shilin He, Wenxiang Jiao, Zhaopeng Tu, Michael Lyu and Xing Wang
• Abstract summary: Non-Autoregressive machine Translation (NAT) models have demonstrated significant inference speedup but suffer from inferior translation accuracy. We propose to adopt Multi-Task learning to transfer the Autoregressive machine Translation knowledge to NAT models through encoder sharing. Experimental results on WMT14 English-German and WMT16 English-Romanian datasets show that the proposed Multi-Task NAT achieves significant improvements over the baseline NAT models.
• Score: 32.77372312124259
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Non-Autoregressive machine Translation (NAT) models have demonstrated significant inference speedup but suffer from inferior translation accuracy. The common practice to tackle the problem is transferring the Autoregressive machine Translation (AT) knowledge to NAT models, e.g., with knowledge distillation. In this work, we hypothesize and empirically verify that AT and NAT encoders capture different linguistic properties of source sentences. Therefore, we propose to adopt Multi-Task learning to transfer the AT knowledge to NAT models through encoder sharing. Specifically, we take the AT model as an auxiliary task to enhance NAT model performance. Experimental results on WMT14 English-German and WMT16 English-Romanian datasets show that the proposed Multi-Task NAT achieves significant improvements over the baseline NAT models. Furthermore, the performance on large-scale WMT19 and WMT20 English-German datasets confirm the consistency of our proposed method. In addition, experimental results demonstrate that our Multi-Task NAT is complementary to knowledge distillation, the standard knowledge transfer method for NAT.

Related papers

• Revisiting Non-Autoregressive Translation at Scale [76.93869248715664]
  We systematically study the impact of scaling on non-autoregressive translation (NAT) behaviors. We show that scaling can alleviate the commonly-cited weaknesses of NAT models, resulting in better translation performance. We establish a new benchmark by validating scaled NAT models on a scaled dataset.
  arXiv  Detail & Related papers  (2023-05-25T15:22:47Z)
• Optimizing Non-Autoregressive Transformers with Contrastive Learning [74.46714706658517]
  Non-autoregressive Transformers (NATs) reduce the inference latency of Autoregressive Transformers (ATs) by predicting words all at once rather than in sequential order. In this paper, we propose to ease the difficulty of modality learning via sampling from the model distribution instead of the data distribution.
  arXiv  Detail & Related papers  (2023-05-23T04:20:13Z)
• Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation [34.22251326493591]
  The Non-Autoregressive Transformer (NAT) achieves great success in neural machine translation tasks. Existing knowledge distillation has side effects, such as propagating errors from the teacher to NAT students. We introduce selective knowledge distillation by introducing an NAT to select NAT-friendly targets that are of high quality and easy to learn.
  arXiv  Detail & Related papers  (2023-03-31T09:16:13Z)
• RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer [15.616188012177538]
  Non-autoregressive neural machine translation (NAT) models are proposed to accelerate the inference process while maintaining relatively high performance. Existing NAT models are difficult to achieve the desired efficiency-quality trade-off. We propose RenewNAT, a flexible framework with high efficiency and effectiveness.
  arXiv  Detail & Related papers  (2023-03-14T07:10:03Z)
• Rephrasing the Reference for Non-Autoregressive Machine Translation [37.816198073720614]
  Non-autoregressive neural machine translation (NAT) models suffer from the multi-modality problem that there may exist multiple possible translations of a source sentence. We introduce a rephraser to provide a better training target for NAT by rephrasing the reference sentence according to the NAT output. Our best variant achieves comparable performance to the autoregressive Transformer, while being 14.7 times more efficient in inference.
  arXiv  Detail & Related papers  (2022-11-30T10:05:03Z)
• Progressive Multi-Granularity Training for Non-Autoregressive Translation [98.11249019844281]
  Non-autoregressive translation (NAT) significantly accelerates the inference process via predicting the entire target sequence. Recent studies show that NAT is weak at learning high-mode of knowledge such as one-to-many translations. We argue that modes can be divided into various granularities which can be learned from easy to hard.
  arXiv  Detail & Related papers  (2021-06-10T07:16:07Z)
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade [47.97977478431973]
  Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks. In this work, we target on closing the performance gap while maintaining the latency advantage.
  arXiv  Detail & Related papers  (2020-12-31T18:52:59Z)
• Understanding and Improving Lexical Choice in Non-Autoregressive Translation [98.11249019844281]
  We propose to expose the raw data to NAT models to restore the useful information of low-frequency words. Our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively.
  arXiv  Detail & Related papers  (2020-12-29T03:18:50Z)
• Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation [188.3605563567253]
  Non-autoregressive translation (NAT) achieves faster inference speed but at the cost of worse accuracy compared with autoregressive translation (AT) We introduce semi-autoregressive translation (SAT) as intermediate tasks. SAT covers AT and NAT as its special cases. We design curriculum schedules to gradually shift k from 1 to N, with different pacing functions and number of tasks trained at the same time. Experiments on IWSLT14 De-En, IWSLT16 En-De, WMT14 En-De and De-En datasets show that TCL-NAT achieves significant accuracy improvements over previous NAT baseline
  arXiv  Detail & Related papers  (2020-07-17T06:06:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.