Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation

• URL: http://arxiv.org/abs/2412.02101v1
• Date: Tue, 03 Dec 2024 02:52:14 GMT
• Title: Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation
• Authors: Zhi Qu, Yiran Wang, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Taro Watanabe,
• Abstract summary: We investigate the issue of the decoder-only architecture to its lack of language transfer capability. We propose dividing the decoding process into two stages so that target tokens are explicitly excluded in the first stage. We impose contrastive learning on translation instructions, resulting in improved performance in zero-shot translation.
• Score: 28.07831604833682
• License:
• Abstract: Existing multilingual neural machine translation (MNMT) approaches mainly focus on improving models with the encoder-decoder architecture to translate multiple languages. However, decoder-only architecture has been explored less in MNMT due to its underperformance when trained on parallel data solely. In this work, we attribute the issue of the decoder-only architecture to its lack of language transfer capability. Specifically, the decoder-only architecture is insufficient in encoding source tokens with the target language features. We propose dividing the decoding process into two stages so that target tokens are explicitly excluded in the first stage to implicitly boost the transfer capability across languages. Additionally, we impose contrastive learning on translation instructions, resulting in improved performance in zero-shot translation. We conduct experiments on TED-19 and OPUS-100 datasets, considering both training from scratch and fine-tuning scenarios. Experimental results show that, compared to the encoder-decoder architecture, our methods not only perform competitively in supervised translations but also achieve improvements of up to 3.39 BLEU, 6.99 chrF++, 3.22 BERTScore, and 4.81 COMET in zero-shot translations.

Related papers

• Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation [16.368747052909214]
  We introduce the identity pair, a sentence translated into itself, to address the lack of the base measure in multilingual investigations. We demonstrate that the encoder transfers the source language to the representational subspace of the target language instead of the language-agnostic state. Based on our findings, we propose two methods: 1) low-rank language-specific embedding at the encoder, and 2) language-specific contrastive learning of the representation at the decoder.
  arXiv  Detail & Related papers  (2024-06-12T11:16:30Z)
• Is Encoder-Decoder Redundant for Neural Machine Translation? [44.37101354412253]
  encoder-decoder architecture is still the de facto neural network architecture for state-of-the-art models. In this work, we experiment with bilingual translation, translation with additional target monolingual data, and multilingual translation. This alternative approach performs on par with the baseline encoder-decoder Transformer, suggesting that an encoder-decoder architecture might be redundant for neural machine translation.
  arXiv  Detail & Related papers  (2022-10-21T08:33:55Z)
• Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders [77.2101943305862]
  We propose a deep encoder with multiple shallow decoders (DEMSD) where each shallow decoder is responsible for a disjoint subset of target languages. DEMSD model with 2-layer decoders is able to obtain a 1.8x speedup on average compared to a standard transformer model with no drop in translation quality.
  arXiv  Detail & Related papers  (2022-06-05T01:15:04Z)
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders [92.90543340071007]
  We introduce DeltaLM, a pretrained multilingual encoder-decoder model. Specifically, we augment the pretrained multilingual encoder with a decoder and pre-train it in a self-supervised way. Experiments show that DeltaLM outperforms various strong baselines on both natural language generation and translation tasks.
  arXiv  Detail & Related papers  (2021-06-25T16:12:10Z)
• Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders [74.89326277221072]
  How to improve the cross-lingual transfer of NMT model with multilingual pretrained encoder is under-explored. We propose SixT, a simple yet effective model for this task. Our model achieves better performance on many-to-English testsets than CRISS and m2m-100.
  arXiv  Detail & Related papers  (2021-04-18T07:42:45Z)
• Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders [19.44855809470709]
  Recently, universal neural machine translation (NMT) with shared encoder-decoder gained good performance on zero-shot translation. Unlike universal NMT, jointly trained language-specific encoders-decoders aim to achieve universal representation across non-shared modules. We study zero-shot translation using language-specific encoders-decoders.
  arXiv  Detail & Related papers  (2021-02-12T15:36:33Z)
• Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation [71.54816893482457]
  We introduce dual-decoder Transformer, a new model architecture that jointly performs automatic speech recognition (ASR) and multilingual speech translation (ST) Our models are based on the original Transformer architecture but consist of two decoders, each responsible for one task (ASR or ST)
  arXiv  Detail & Related papers  (2020-11-02T04:59:50Z)
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation [81.7786241489002]
  Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics. We propose random online backtranslation to enforce the translation of unseen training language pairs.
  arXiv  Detail & Related papers  (2020-04-24T17:21:32Z)
• Bi-Decoder Augmented Network for Neural Machine Translation [108.3931242633331]
  We propose a novel Bi-Decoder Augmented Network (BiDAN) for the neural machine translation task. Since each decoder transforms the representations of the input text into its corresponding language, jointly training with two target ends can make the shared encoder has the potential to produce a language-independent semantic space.
  arXiv  Detail & Related papers  (2020-01-14T02:05:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.