Optimizing Transformer for Low-Resource Neural Machine Translation

• URL: http://arxiv.org/abs/2011.02266v1
• Date: Wed, 4 Nov 2020 13:12:29 GMT
• Title: Optimizing Transformer for Low-Resource Neural Machine Translation
• Authors: Ali Araabi, Christof Monz
• Abstract summary: Language pairs with limited amounts of parallel data, also known as low-resource languages, remain a challenge for neural machine translation. Our experiments on different subsets of the IWSLT14 training data show that the effectiveness of Transformer under low-resource conditions is highly dependent on the hyper- parameter settings. Using an optimized Transformer for low-resource conditions improves the translation quality up to 7.3 BLEU points compared to using the Transformer default settings.
• Score: 4.802292434636455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language pairs with limited amounts of parallel data, also known as low-resource languages, remain a challenge for neural machine translation. While the Transformer model has achieved significant improvements for many language pairs and has become the de facto mainstream architecture, its capability under low-resource conditions has not been fully investigated yet. Our experiments on different subsets of the IWSLT14 training data show that the effectiveness of Transformer under low-resource conditions is highly dependent on the hyper-parameter settings. Our experiments show that using an optimized Transformer for low-resource conditions improves the translation quality up to 7.3 BLEU points compared to using the Transformer default settings.

Related papers

• Transformers for Low-Resource Languages:Is F\'eidir Linn! [2.648836772989769]
  In general, neural translation models often under perform on language pairs with insufficient training data. We demonstrate that choosing appropriate parameters leads to considerable performance improvements. A Transformer optimized model demonstrated a BLEU score improvement of 7.8 points when compared with a baseline RNN model.
  arXiv  Detail & Related papers  (2024-03-04T12:29:59Z)
• Enhancing Neural Machine Translation of Low-Resource Languages: Corpus Development, Human Evaluation and Explainable AI Architectures [0.0]
  The Transformer architecture stands out as the gold standard, especially for high-resource language pairs. The scarcity of parallel datasets for low-resource languages can hinder machine translation development. This thesis introduces adaptNMT and adaptMLLM, two open-source applications streamlined for the development, fine-tuning, and deployment of neural machine translation models.
  arXiv  Detail & Related papers  (2024-03-03T18:08:30Z)
• Quick Back-Translation for Unsupervised Machine Translation [9.51657235413336]
  We propose a two-for-one improvement to Transformer back-translation: Quick Back-Translation (QBT) QBT re-purposes the encoder as a generative model, and uses encoder-generated sequences to train the decoder. Experiments on various WMT benchmarks demonstrate that QBT dramatically outperforms standard back-translation only method in terms of training efficiency.
  arXiv  Detail & Related papers  (2023-12-01T20:27:42Z)
• Enhanced Transformer Architecture for Natural Language Processing [2.6071653283020915]
  Transformer is a state-of-the-art model in the field of natural language processing (NLP) In this paper, a novel structure of Transformer is proposed. It is featured by full layer normalization, weighted residual connection, positional encoding exploiting reinforcement learning, and zero masked self-attention. The proposed Transformer model, which is called Enhanced Transformer, is validated by the bilingual evaluation understudy (BLEU) score obtained with the Multi30k translation dataset.
  arXiv  Detail & Related papers  (2023-10-17T01:59:07Z)
• Strategies for improving low resource speech to text translation relying on pre-trained ASR models [59.90106959717875]
  This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST) We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively.
  arXiv  Detail & Related papers  (2023-05-31T21:58:07Z)
• Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation [9.294853905247383]
  We study the use of deep Transformer translation model for the CCMT 2022 Chinese-Thai low-resource machine translation task. Considering that increasing the number of layers also increases the regularization on new model parameters, we adopt the highest performance setting but increase the depth of the Transformer to 24 layers. Our work obtains the SOTA performance in the Chinese-to-Thai translation in the constrained evaluation.
  arXiv  Detail & Related papers  (2022-12-24T05:35:04Z)
• Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks [126.33843752332139]
  We introduce Group-wise Transformation towards a universal yet lightweight Transformer for vision-and-language tasks, termed as LW-Transformer. We apply LW-Transformer to a set of Transformer-based networks, and quantitatively measure them on three vision-and-language tasks and six benchmark datasets. Experimental results show that while saving a large number of parameters and computations, LW-Transformer achieves very competitive performance against the original Transformer networks for vision-and-language tasks.
  arXiv  Detail & Related papers  (2022-04-16T11:30:26Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)
• Applying the Transformer to Character-level Transduction [68.91664610425114]
  The transformer has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks. We show that with a large enough batch size, the transformer does indeed outperform recurrent models for character-level tasks.
  arXiv  Detail & Related papers  (2020-05-20T17:25:43Z)
• Explicit Reordering for Neural Machine Translation [50.70683739103066]
  In Transformer-based neural machine translation (NMT), the positional encoding mechanism helps the self-attention networks to learn the source representation with order dependency. We propose a novel reordering method to explicitly model this reordering information for the Transformer-based NMT. The empirical results on the WMT14 English-to-German, WAT ASPEC Japanese-to-English, and WMT17 Chinese-to-English translation tasks show the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2020-04-08T05:28:46Z)
• Transformer on a Diet [81.09119185568296]
  Transformer has been widely used thanks to its ability to capture sequence information in an efficient way. Recent developments, such as BERT and GPT-2, deliver only heavy architectures with a focus on effectiveness. We explore three carefully-designed light Transformer architectures to figure out whether the Transformer with less computations could produce competitive results.
  arXiv  Detail & Related papers  (2020-02-14T18:41:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.