On the Shortcut Learning in Multilingual Neural Machine Translation

• URL: http://arxiv.org/abs/2411.10581v1
• Date: Fri, 15 Nov 2024 21:09:36 GMT
• Title: On the Shortcut Learning in Multilingual Neural Machine Translation
• Authors: Wenxuan Wang, Wenxiang Jiao, Jen-tse Huang, Zhaopeng Tu, Michael R. Lyu,
• Abstract summary: This study revisits the commonly-cited off-target issue in multilingual neural machine translation (MNMT) We attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings. Analyses on learning dynamics show that the shortcut learning generally occurs in the later stage of model training.
• Score: 95.30470845501141
• License:
• Abstract: In this study, we revisit the commonly-cited off-target issue in multilingual neural machine translation (MNMT). By carefully designing experiments on different MNMT scenarios and models, we attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings. Specifically, the learned shortcuts biases MNMT to mistakenly translate non-centric languages into the centric language instead of the expected non-centric language for zero-shot translation. Analyses on learning dynamics show that the shortcut learning generally occurs in the later stage of model training, and multilingual pretraining accelerates and aggravates the shortcut learning. Based on these observations, we propose a simple and effective training strategy to eliminate the shortcuts in MNMT models by leveraging the forgetting nature of model training. The only difference from the standard training is that we remove the training instances that may induce the shortcut learning in the later stage of model training. Without introducing any additional data and computational costs, our approach can consistently and significantly improve the zero-shot translation performance by alleviating the shortcut learning for different MNMT models and benchmarks.

Related papers

• Extending Multilingual Machine Translation through Imitation Learning [60.15671816513614]
  Imit-MNMT treats the task as an imitation learning process, which mimicks the behavior of an expert. We show that our approach significantly improves the translation performance between the new and the original languages. We also demonstrate that our approach is capable of solving copy and off-target problems.
  arXiv  Detail & Related papers  (2023-11-14T21:04:03Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT [64.1841519527504]
  neural machine translation uses a single neural network to model the entire translation process. Despite neural machine translation being de-facto standard, it is still not clear how NMT models acquire different competences over the course of training.
  arXiv  Detail & Related papers  (2021-09-03T09:38:50Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)
• Token-wise Curriculum Learning for Neural Machine Translation [94.93133801641707]
  Existing curriculum learning approaches to Neural Machine Translation (NMT) require sufficient sampling amounts of "easy" samples from training data at the early training stage. We propose a novel token-wise curriculum learning approach that creates sufficient amounts of easy samples. Our approach can consistently outperform baselines on 5 language pairs, especially for low-resource languages.
  arXiv  Detail & Related papers  (2021-03-20T03:57:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.