Bilingual Dictionary-based Language Model Pretraining for Neural Machine Translation

• URL: http://arxiv.org/abs/2103.07040v1
• Date: Fri, 12 Mar 2021 02:01:22 GMT
• Title: Bilingual Dictionary-based Language Model Pretraining for Neural Machine Translation
• Authors: Yusen Lin, Jiayong Lin, Shuaicheng Zhang, Haoying Dai
• Abstract summary: We incorporate the translation information from dictionaries into the pretraining process and propose a novel Bilingual Dictionary-based Language Model (BDLM) We evaluate our BDLM in Chinese, English, and Romanian.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent studies have demonstrated a perceivable improvement on the performance of neural machine translation by applying cross-lingual language model pretraining (Lample and Conneau, 2019), especially the Translation Language Modeling (TLM). To alleviate the need for expensive parallel corpora by TLM, in this work, we incorporate the translation information from dictionaries into the pretraining process and propose a novel Bilingual Dictionary-based Language Model (BDLM). We evaluate our BDLM in Chinese, English, and Romanian. For Chinese-English, we obtained a 55.0 BLEU on WMT-News19 (Tiedemann, 2012) and a 24.3 BLEU on WMT20 news-commentary, outperforming the Vanilla Transformer (Vaswani et al., 2017) by more than 8.4 BLEU and 2.3 BLEU, respectively. According to our results, the BDLM also has advantages on convergence speed and predicting rare words. The increase in BLEU for WMT16 Romanian-English also shows its effectiveness in low-resources language translation.

Related papers

• NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Models [2.186901738997927]
  This paper introduces NusaMT-7B, an LLM-based machine translation model for low-resource Indonesian languages. Our approach integrates continued pre-training on monolingual data,Supervised Fine-Tuning (SFT), self-learning, and an LLM-based data cleaner to reduce noise in parallel sentences. Our results show that fine-tuned LLMs can enhance translation quality for low-resource languages, aiding in linguistic preservation and cross-cultural communication.
  arXiv  Detail & Related papers  (2024-10-10T11:33:25Z)
• Machine Translation Advancements of Low-Resource Indian Languages by Transfer Learning [9.373815852241648]
  We employ two distinct knowledge transfer strategies to develop a reliable machine translation system for low-resource Indian languages. For Assamese(as) and Manipuri(mn), we fine-tuned the existing IndicTrans2 open-source model to enable bidirectional translation between English and these languages. For Khasi (kh) and Mizo (mz), We trained a multilingual model as a baseline using bilingual data from these four language pairs, along with an additional about 8kw English-Bengali bilingual data.
  arXiv  Detail & Related papers  (2024-09-24T08:53:19Z)
• Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language [1.1702440973773898]
  This study explores the use of large language models for translating English into Mambai, a low-resource Austronesian language spoken in Timor-Leste. Our methodology involves the strategic selection of parallel sentences and dictionary entries for prompting. We find that including dictionary entries in prompts and a mix of sentences retrieved through-IDF and semantic embeddings significantly improves translation quality.
  arXiv  Detail & Related papers  (2024-04-07T05:04:38Z)
• Machine Translation for Ge'ez Language [0.0]
  Machine translation for low-resource languages such as Ge'ez faces challenges such as out-of-vocabulary words, domain mismatches, and lack of labeled training data. We develop a multilingual neural machine translation (MNMT) model based on languages relatedness. We also experiment with using GPT-3.5, a state-of-the-art LLM, for few-shot translation with fuzzy matches.
  arXiv  Detail & Related papers  (2023-11-24T14:55:23Z)
• Chain-of-Dictionary Prompting Elicits Translation in Large Language Models [100.47154959254937]
  Large language models (LLMs) have shown surprisingly good performance in multilingual neural machine translation (MNMT) We present a novel method, CoD, which augments LLMs with prior knowledge with the chains of multilingual dictionaries for a subset of input words to elicit translation abilities.
  arXiv  Detail & Related papers  (2023-05-11T05:19:47Z)
• Tencent AI Lab - Shanghai Jiao Tong University Low-Resource Translation System for the WMT22 Translation Task [49.916963624249355]
  This paper describes Tencent AI Lab - Shanghai Jiao Tong University (TAL-SJTU) Low-Resource Translation systems for the WMT22 shared task. We participate in the general translation task on English$Leftrightarrow$Livonian. Our system is based on M2M100 with novel techniques that adapt it to the target language pair.
  arXiv  Detail & Related papers  (2022-10-17T04:34:09Z)
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation [53.22775597051498]
  We present a continual pre-training framework on mBART to effectively adapt it to unseen languages. Results show that our method can consistently improve the fine-tuning performance upon the mBART baseline. Our approach also boosts the performance on translation pairs where both languages are seen in the original mBART's pre-training.
  arXiv  Detail & Related papers  (2021-05-09T14:49:07Z)
• Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation [127.81351683335143]
  Cross-lingual pretraining requires models to align the lexical- and high-level representations of the two languages. Previous research has shown that this is because the representations are not sufficiently aligned. In this paper, we enhance the bilingual masked language model pretraining with lexical-level information by using type-level cross-lingual subword embeddings.
  arXiv  Detail & Related papers  (2021-03-18T21:17:58Z)
• Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings [72.69253034282035]
  We exploit a language independent multilingual sentence representation to easily generalize to a new language. Blindly decoding from Portuguese using a basesystem containing several Romance languages we achieve scores of 36.4 BLEU for Portuguese-English and 12.8 BLEU for Russian-English. We explore a more practical adaptation approach through non-iterative backtranslation, exploiting our model's ability to produce high quality translations.
  arXiv  Detail & Related papers  (2021-03-11T14:22:08Z)
• Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT [129.99918589405675]
  We present an effective approach that reuses an LM that is pretrained only on the high-resource language. The monolingual LM is fine-tuned on both languages and is then used to initialize a UNMT model. Our approach, RE-LM, outperforms a competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk) and English-Albanian (En-Sq)
  arXiv  Detail & Related papers  (2020-09-16T11:37:10Z)
• Neural Machine Translation for Low-Resourced Indian Languages [4.726777092009554]
  Machine translation is an effective approach to convert text to a different language without any human involvement. In this paper, we have applied NMT on two of the most morphological rich Indian languages, i.e. English-Tamil and English-Malayalam. We proposed a novel NMT model using Multihead self-attention along with pre-trained Byte-Pair-Encoded (BPE) and MultiBPE embeddings to develop an efficient translation system.
  arXiv  Detail & Related papers  (2020-04-19T17:29:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.