Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation

• URL: http://arxiv.org/abs/2503.06594v1
• Date: Sun, 09 Mar 2025 12:54:05 GMT
• Title: Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
• Authors: Yingfeng Luo, Tong Zheng, Yongyu Mu, Bei Li, Qinghong Zhang, Yongqi Gao, Ziqiang Xu, Peinan Feng, Xiaoqian Liu, Tong Xiao, Jingbo Zhu,
• Abstract summary: We explore translation models that are universal, efficient, and easy to optimize.<n>We apply large language models (LLMs) to NMT encoding and leave the NMT decoder unchanged.<n>We construct a new dataset involving multiple tasks to assess how well the machine translation system generalizes.
• Score: 40.72168378706009
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The field of neural machine translation (NMT) has changed with the advent of large language models (LLMs). Much of the recent emphasis in natural language processing (NLP) has been on modeling machine translation and many other problems using a single pre-trained Transformer decoder, while encoder-decoder architectures, which were the standard in earlier NMT models, have received relatively less attention. In this paper, we explore translation models that are universal, efficient, and easy to optimize, by marrying the world of LLMs with the world of NMT. We apply LLMs to NMT encoding and leave the NMT decoder unchanged. We also develop methods for adapting LLMs to work better with the NMT decoder. Furthermore, we construct a new dataset involving multiple tasks to assess how well the machine translation system generalizes across various tasks. Evaluations on the WMT and our datasets show that results using our method match or surpass a range of baselines in terms of translation quality, but achieve $2.4 \sim 6.5 \times$ inference speedups and a $75\%$ reduction in the memory footprint of the KV cache. It also demonstrates strong generalization across a variety of translation-related tasks.

Related papers

• Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding [73.32763904267186]
  Large Language Models (LLMs) present the potential for achieving superior translation quality. We propose Cooperative Decoding (CoDec) which treats NMT systems as a pretranslation model and MT-oriented LLMs as a supplemental solution.
  arXiv  Detail & Related papers  (2023-11-06T03:41:57Z)
• Simultaneous Machine Translation with Large Language Models [51.470478122113356]
  We investigate the possibility of applying Large Language Models to SimulMT tasks. We conducted experiments using the textttLlama2-7b-chat model on nine different languages from the MUST-C dataset. The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics.
  arXiv  Detail & Related papers  (2023-09-13T04:06:47Z)
• Language Models are Good Translators [63.528370845657896]
  We show that a single language model (LM4MT) can achieve comparable performance with strong encoder-decoder NMT models. Experiments on pivot-based and zero-shot translation tasks show that LM4MT can outperform the encoder-decoder NMT model by a large margin.
  arXiv  Detail & Related papers  (2021-06-25T13:30:29Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)
• Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders [74.89326277221072]
  How to improve the cross-lingual transfer of NMT model with multilingual pretrained encoder is under-explored. We propose SixT, a simple yet effective model for this task. Our model achieves better performance on many-to-English testsets than CRISS and m2m-100.
  arXiv  Detail & Related papers  (2021-04-18T07:42:45Z)
• Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders [19.44855809470709]
  Recently, universal neural machine translation (NMT) with shared encoder-decoder gained good performance on zero-shot translation. Unlike universal NMT, jointly trained language-specific encoders-decoders aim to achieve universal representation across non-shared modules. We study zero-shot translation using language-specific encoders-decoders.
  arXiv  Detail & Related papers  (2021-02-12T15:36:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.