LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation

• URL: http://arxiv.org/abs/2406.02876v2
• Date: Thu, 6 Jun 2024 03:22:38 GMT
• Title: LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation
• Authors: Zengkui Sun, Yijin Liu, Fandong Meng, Jinan Xu, Yufeng Chen, Jie Zhou,
• Abstract summary: We propose a simple yet effective strategy named Language Converter Strategy (LCS) By introducing the target language embedding into the top encoder layers, LCS mitigates confusion in the encoder and ensures stable language indication for the decoder. Experimental results on MultiUN, TED, and OPUS-100 datasets demonstrate that LCS could significantly mitigate the off-target issue.
• Score: 84.38105530043741
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilingual neural machine translation models generally distinguish translation directions by the language tag (LT) in front of the source or target sentences. However, current LT strategies cannot indicate the desired target language as expected on zero-shot translation, i.e., the off-target issue. Our analysis reveals that the indication of the target language is sensitive to the placement of the target LT. For example, when placing the target LT on the decoder side, the indication would rapidly degrade along with decoding steps, while placing the target LT on the encoder side would lead to copying or paraphrasing the source input. To address the above issues, we propose a simple yet effective strategy named Language Converter Strategy (LCS). By introducing the target language embedding into the top encoder layers, LCS mitigates confusion in the encoder and ensures stable language indication for the decoder. Experimental results on MultiUN, TED, and OPUS-100 datasets demonstrate that LCS could significantly mitigate the off-target issue, with language accuracy up to 95.28%, 96.21%, and 85.35% meanwhile outperforming the vanilla LT strategy by 3.07, 3,3, and 7.93 BLEU scores on zero-shot translation, respectively.

Related papers

• Language-Informed Beam Search Decoding for Multilingual Machine Translation [24.044315362087687]
  Language-informed Beam Search (LiBS) is a general decoding algorithm incorporating an off-the-shelf Language Identification (LiD) model into beam search decoding to reduce off-target translations. Results show that our proposed LiBS algorithm on average improves +1.1 BLEU and +0.9 BLEU on WMT and OPUS datasets, and reduces off-target rates from 22.9% to 7.7% and 65.8% to 25.3% respectively.
  arXiv  Detail & Related papers  (2024-08-11T09:57:46Z)
• Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation [16.368747052909214]
  We introduce the identity pair, a sentence translated into itself, to address the lack of the base measure in multilingual investigations. We demonstrate that the encoder transfers the source language to the representational subspace of the target language instead of the language-agnostic state. Based on our findings, we propose two methods: 1) low-rank language-specific embedding at the encoder, and 2) language-specific contrastive learning of the representation at the decoder.
  arXiv  Detail & Related papers  (2024-06-12T11:16:30Z)
• Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning [57.323716555996114]
  Off-target translation remains an unsolved problem, especially for low-resource languages. Recent works have either designed advanced prompting strategies to highlight the functionality of translation instructions or exploited the in-context learning ability of LLMs. In this work, we design a two-stage fine-tuning algorithm to improve the instruction-following ability (especially the translation direction) of LLMs.
  arXiv  Detail & Related papers  (2024-03-21T13:47:40Z)
• Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation [79.96416609433724]
  Zero-shot translation (ZST) aims to translate between unseen language pairs in training data. The common practice to guide the zero-shot language mapping during inference is to deliberately insert the source and target language IDs. Recent studies have shown that language IDs sometimes fail to navigate the ZST task, making them suffer from the off-target problem.
  arXiv  Detail & Related papers  (2023-09-28T17:02:36Z)
• On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation [104.85258654917297]
  We find that failing in encoding discriminative target language signal will lead to off-target and a closer lexical distance. We propose Language Aware Vocabulary Sharing (LAVS) to construct the multilingual vocabulary. We conduct experiments on a multilingual machine translation benchmark in 11 languages.
  arXiv  Detail & Related papers  (2023-05-18T12:43:31Z)
• Improving Multilingual Translation by Representation and Gradient Regularization [82.42760103045083]
  We propose a joint approach to regularize NMT models at both representation-level and gradient-level. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance.
  arXiv  Detail & Related papers  (2021-09-10T10:52:21Z)
• Language Tags Matter for Zero-Shot Neural Machine Translation [17.353423698436547]
  Language tag (LT) strategies are often adopted to indicate the translation directions in MNMT. We demonstrate that the LTs are not only indicators for translation directions but also crucial to zero-shot translation qualities. Experimental results show that by ignoring the source language tag (SLT) and adding the target language tag (TLT) to the encoder, the zero-shot translations could achieve a +8 BLEU score difference.
  arXiv  Detail & Related papers  (2021-06-15T07:32:36Z)
• Improving Target-side Lexical Transfer in Multilingual Neural Machine Translation [104.10726545151043]
  multilingual data has been found more beneficial for NMT models that translate from the LRL to a target language than the ones that translate into the LRLs. Our experiments show that DecSDE leads to consistent gains of up to 1.8 BLEU on translation from English to four different languages.
  arXiv  Detail & Related papers  (2020-10-04T19:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.