Character-level Transformer-based Neural Machine Translation

• URL: http://arxiv.org/abs/2005.11239v1
• Date: Fri, 22 May 2020 15:40:43 GMT
• Title: Character-level Transformer-based Neural Machine Translation
• Authors: Nikolay Banar, Walter Daelemans and Mike Kestemont
• Abstract summary: We discuss a novel, Transformer-based approach, that we compare, both in speed and in quality to the Transformer at subword and character levels. We evaluate our models on 4 language pairs from WMT'15: DE-EN, CS-EN, FI-EN and RU-EN. The proposed novel architecture can be trained on a single GPU and is 34% percent faster than the character-level Transformer.
• Score: 5.699756532377753
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural machine translation (NMT) is nowadays commonly applied at the subword level, using byte-pair encoding. A promising alternative approach focuses on character-level translation, which simplifies processing pipelines in NMT considerably. This approach, however, must consider relatively longer sequences, rendering the training process prohibitively expensive. In this paper, we discuss a novel, Transformer-based approach, that we compare, both in speed and in quality to the Transformer at subword and character levels, as well as previously developed character-level models. We evaluate our models on 4 language pairs from WMT'15: DE-EN, CS-EN, FI-EN and RU-EN. The proposed novel architecture can be trained on a single GPU and is 34% percent faster than the character-level Transformer; still, the obtained results are at least on par with it. In addition, our proposed model outperforms the subword-level model in FI-EN and shows close results in CS-EN. To stimulate further research in this area and close the gap with subword-level NMT, we make all our code and models publicly available.

Related papers

• Character-level NMT and language similarity [1.90365714903665]
  We explore the effectiveness of character-level neural machine translation for various levels of language similarity and size of the training dataset on translation between Czech and Croatian, German, Hungarian, Slovak, and Spanish. We evaluate the models using automatic MT metrics and show that translation between similar languages benefits from character-level input segmentation. We confirm previous findings that it is possible to close the gap by finetuning the already trained subword-level models to character-level.
  arXiv  Detail & Related papers  (2023-08-08T17:01:42Z)
• Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation [9.736284584478032]
  We show the effectiveness of character-level modeling in translation, particularly in cases where fine-tuning data is limited. While evaluating the importance of source texts in driving model predictions, we highlight word-level patterns within ByT5. We conclude by assessing the efficiency tradeoff of byte models, suggesting their usage in non-time-critical scenarios to boost translation quality.
  arXiv  Detail & Related papers  (2023-02-28T00:50:19Z)
• A Template-based Method for Constrained Neural Machine Translation [100.02590022551718]
  We propose a template-based method that can yield results with high translation quality and match accuracy while keeping the decoding speed. The generation and derivation of the template can be learned through one sequence-to-sequence training framework. Experimental results show that the proposed template-based methods can outperform several representative baselines in lexically and structurally constrained translation tasks.
  arXiv  Detail & Related papers  (2022-05-23T12:24:34Z)
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization [50.16128796194463]
  We propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. We introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level.
  arXiv  Detail & Related papers  (2021-06-23T22:24:14Z)
• Revisiting Simple Neural Probabilistic Language Models [27.957834093475686]
  This paper revisits the neural probabilistic language model (NPLM) ofcitetBengio2003ANP. When scaled up to modern hardware, this model performs much better than expected on word-level language model benchmarks. Inspired by this result, we modify the Transformer by replacing its first self-attention layer with the NPLM's local concatenation layer.
  arXiv  Detail & Related papers  (2021-04-08T02:18:47Z)
• Rethinking Document-level Neural Machine Translation [73.42052953710605]
  We try to answer the question: Is the capacity of current models strong enough for document-level translation? We observe that the original Transformer with appropriate training techniques can achieve strong results for document translation, even with a length of 2000 words.
  arXiv  Detail & Related papers  (2020-10-18T11:18:29Z)
• Learning Source Phrase Representations for Neural Machine Translation [65.94387047871648]
  We propose an attentive phrase representation generation mechanism which is able to generate phrase representations from corresponding token representations. In our experiments, we obtain significant improvements on the WMT 14 English-German and English-French tasks on top of the strong Transformer baseline.
  arXiv  Detail & Related papers  (2020-06-25T13:43:11Z)
• Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems [78.80826533405019]
  We show that we can obtain a neural machine translation model that works at the character level without requiring token segmentation. Our study is a significant step towards high-performance and easy to train character-based models that are not extremely large.
  arXiv  Detail & Related papers  (2020-04-29T15:56:02Z)
• Explicit Reordering for Neural Machine Translation [50.70683739103066]
  In Transformer-based neural machine translation (NMT), the positional encoding mechanism helps the self-attention networks to learn the source representation with order dependency. We propose a novel reordering method to explicitly model this reordering information for the Transformer-based NMT. The empirical results on the WMT14 English-to-German, WAT ASPEC Japanese-to-English, and WMT17 Chinese-to-English translation tasks show the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2020-04-08T05:28:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.