Machine Translation of Novels in the Age of Transformer

• URL: http://arxiv.org/abs/2011.14979v1
• Date: Mon, 30 Nov 2020 16:51:08 GMT
• Title: Machine Translation of Novels in the Age of Transformer
• Authors: Antonio Toral, Antoni Oliver, Pau Ribas Ballest\'in
• Abstract summary: We build a machine translation system tailored to the literary domain, specifically to novels, based on the state-of-the-art architecture in neural MT (NMT), the Transformer, for the translation direction English-to-Catalan. We compare this MT system against three other systems (two domain-specific systems under the recurrent and phrase-based paradigms and a popular generic on-line system) on three evaluations. As expected, the domain-specific Transformer-based system outperformed the three other systems in all the three evaluations conducted, in all cases by a large margin.
• Score: 1.6453685972661827
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this chapter we build a machine translation (MT) system tailored to the literary domain, specifically to novels, based on the state-of-the-art architecture in neural MT (NMT), the Transformer (Vaswani et al., 2017), for the translation direction English-to-Catalan. Subsequently, we assess to what extent such a system can be useful by evaluating its translations, by comparing this MT system against three other systems (two domain-specific systems under the recurrent and phrase-based paradigms and a popular generic on-line system) on three evaluations. The first evaluation is automatic and uses the most-widely used automatic evaluation metric, BLEU. The two remaining evaluations are manual and they assess, respectively, preference and amount of post-editing required to make the translation error-free. As expected, the domain-specific Transformer-based system outperformed the three other systems in all the three evaluations conducted, in all cases by a large margin.

Related papers

• Evaluating Automatic Metrics with Incremental Machine Translation Systems [55.78547133890403]
  We introduce a dataset comprising commercial machine translations, gathered weekly over six years across 12 translation directions. We assume commercial systems improve over time, which enables us to evaluate machine translation (MT) metrics based on their preference for more recent translations. Our study confirms several previous findings in MT metrics research and demonstrates the dataset's value as a testbed for metric evaluation.
  arXiv  Detail & Related papers  (2024-07-03T17:04:17Z)
• Good, but not always Fair: An Evaluation of Gender Bias for three commercial Machine Translation Systems [4.802214389376064]
  This paper offers a meticulous assessment of three commercial Machine Translation systems - Google Translate, DeepL, and Modern MT. For three language pairs (English/Spanish, English/Italian, and English/French), we scrutinize the behavior of such systems at several levels of granularity and on a variety of naturally occurring gender phenomena in translation. Our study takes stock of the current state of online MT tools, by revealing significant discrepancies in the gender translation of the three systems, with each system displaying varying degrees of bias despite their overall translation quality.
  arXiv  Detail & Related papers  (2023-06-09T13:24:27Z)
• Supervised Visual Attention for Simultaneous Multimodal Machine Translation [47.18251159303909]
  We propose the first Transformer-based simultaneous machine translation (MMT) architecture. We extend this model with an auxiliary supervision signal that guides its visual attention mechanism using labelled phrase-region alignments. Our results show that supervised visual attention consistently improves the translation quality of the MMT models.
  arXiv  Detail & Related papers  (2022-01-23T17:25:57Z)
• Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task [95.06453182273027]
  This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation. Our model submissions to the shared task were with DeltaLMnotefooturlhttps://aka.ms/deltalm, a generic pre-trained multilingual-decoder model. Our final submissions ranked first on three tracks in terms of the automatic evaluation metric.
  arXiv  Detail & Related papers  (2021-11-03T09:16:17Z)
• It is Not as Good as You Think! Evaluating Simultaneous Machine Translation on Interpretation Data [58.105938143865906]
  We argue that SiMT systems should be trained and tested on real interpretation data. Our results highlight the difference of up-to 13.83 BLEU score when SiMT models are evaluated on translation vs interpretation data.
  arXiv  Detail & Related papers  (2021-10-11T12:27:07Z)
• Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort [0.0]
  In Machine Translation, assessing the quality of a large amount of automatic translations can be challenging. We propose a novel application of online learning that, given an ensemble of Machine Translation systems, dynamically converges to the best systems. Our experiments on WMT'19 datasets show that our online approach quickly converges to the top-3 ranked systems for the language pairs considered.
  arXiv  Detail & Related papers  (2021-05-27T18:19:39Z)
• The Impact of Indirect Machine Translation on Sentiment Classification [6.719549885077474]
  We propose employing a machine translation (MT) system to translate customer feedback into another language. As performing a direct translation is not always possible, we explore the performance of automatic classifiers on sentences that have been translated. We conduct several experiments to analyse the performance of our proposed sentiment classification system and discuss the advantages and drawbacks of classifying translated sentences.
  arXiv  Detail & Related papers  (2020-08-25T20:30:21Z)
• Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese [3.3453601632404073]
  We develop an error taxonomy compliant with the Multidimensional Quality Metrics (MQM) framework. We then conduct an error annotation using this customised error taxonomy on the output of state-of-the-art recurrent- and Transformer-based MT systems. The resulting annotation shows that, compared to the best recurrent system, the best Transformer system results in a 31% reduction of the total number of errors.
  arXiv  Detail & Related papers  (2020-06-15T11:47:00Z)
• Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics [64.88815792555451]
  We show that current methods for judging metrics are highly sensitive to the translations used for assessment. We develop a method for thresholding performance improvement under an automatic metric against human judgements.
  arXiv  Detail & Related papers  (2020-06-11T09:12:53Z)
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation [55.02832094101173]
  Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual similarity. This paper concerns ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations.
  arXiv  Detail & Related papers  (2020-05-03T22:10:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.