Rethinking the Reasonability of the Test Set for Simultaneous Machine Translation

• URL: http://arxiv.org/abs/2303.00969v1
• Date: Thu, 2 Mar 2023 05:06:44 GMT
• Title: Rethinking the Reasonability of the Test Set for Simultaneous Machine Translation
• Authors: Mengge Liu, Wen Zhang, Xiang Li, Jian Luan, Bin Wang, Yuhang Guo, Shuoying Chen
• Abstract summary: Simultaneous machine translation (SimulMT) models start translation before the end of the source sentence. General full-sentence translation test set is acquired by offline translation of the entire source sentence. We manually annotate a monotonic test set based on the MuST-C English-Chinese test set, denoted as SiMuST-C.
• Score: 14.758033756564858
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simultaneous machine translation (SimulMT) models start translation before the end of the source sentence, making the translation monotonically aligned with the source sentence. However, the general full-sentence translation test set is acquired by offline translation of the entire source sentence, which is not designed for SimulMT evaluation, making us rethink whether this will underestimate the performance of SimulMT models. In this paper, we manually annotate a monotonic test set based on the MuST-C English-Chinese test set, denoted as SiMuST-C. Our human evaluation confirms the acceptability of our annotated test set. Evaluations on three different SimulMT models verify that the underestimation problem can be alleviated on our test set. Further experiments show that finetuning on an automatically extracted monotonic training set improves SimulMT models by up to 3 BLEU points.

Related papers

• Word Order in English-Japanese Simultaneous Interpretation: Analyses and Evaluation using Chunk-wise Monotonic Translation [13.713981533436135]
  This paper analyzes the features of monotonic translations, which follow the word order of the source language, in simultaneous interpreting (SI) We analyzed the characteristics of chunk-wise monotonic translation (CMT) sentences using the NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation dataset. We further investigated the features of CMT sentences by evaluating the output from the existing speech translation (ST) and simultaneous speech translation (simulST) models on the NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation dataset.
  arXiv  Detail & Related papers  (2024-06-13T09:10:16Z)
• M3ST: Mix at Three Levels for Speech Translation [66.71994367650461]
  We propose Mix at three levels for Speech Translation (M3ST) method to increase the diversity of the augmented training corpus. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. Experiments on MuST-C speech translation benchmark and analysis show that M3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
  arXiv  Detail & Related papers  (2022-12-07T14:22:00Z)
• Improving Simultaneous Machine Translation with Monolingual Data [94.1085601198393]
  Simultaneous machine translation (SiMT) is usually done via sequence-level knowledge distillation (Seq-KD) from a full-sentence neural machine translation (NMT) model. We propose to leverage monolingual data to improve SiMT, which trains a SiMT student on the combination of bilingual data and external monolingual data distilled by Seq-KD.
  arXiv  Detail & Related papers  (2022-12-02T14:13:53Z)
• Anticipation-free Training for Simultaneous Translation [70.85761141178597]
  Simultaneous translation (SimulMT) speeds up the translation process by starting to translate before the source sentence is completely available. Existing methods increase latency or introduce adaptive read-write policies for SimulMT models to handle local reordering and improve translation quality. We propose a new framework that decomposes the translation process into the monotonic translation step and the reordering step.
  arXiv  Detail & Related papers  (2022-01-30T16:29:37Z)
• BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation [53.55009917938002]
  We propose to refine the mined bitexts via automatic editing. Experiments demonstrate that our approach successfully improves the quality of CCMatrix mined bitext for 5 low-resource language-pairs and 10 translation directions by up to 8 BLEU points.
  arXiv  Detail & Related papers  (2021-11-12T16:00:39Z)
• Variance-Aware Machine Translation Test Sets [19.973201669851626]
  We release 70 small and discriminative test sets for machine translation (MT) evaluation called variance-aware test sets (VAT) VAT is automatically created by a novel variance-aware filtering method that filters the indiscriminative test instances of the current MT test sets without any human labor.
  arXiv  Detail & Related papers  (2021-11-07T13:18:59Z)
• It is Not as Good as You Think! Evaluating Simultaneous Machine Translation on Interpretation Data [58.105938143865906]
  We argue that SiMT systems should be trained and tested on real interpretation data. Our results highlight the difference of up-to 13.83 BLEU score when SiMT models are evaluated on translation vs interpretation data.
  arXiv  Detail & Related papers  (2021-10-11T12:27:07Z)
• ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback [70.5469946314539]
  ChrEnTranslate is an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability.
  arXiv  Detail & Related papers  (2021-07-30T17:58:54Z)
• Translating the Unseen? Yor\`ub\'a $\rightarrow$ English MT in Low-Resource, Morphologically-Unmarked Settings [8.006185289499049]
  Translating between languages where certain features are marked morphologically in one but absent or marked contextually in the other is an important test case for machine translation. In this work, we perform fine-grained analysis on how an SMT system compares with two NMT systems when translating bare nouns in Yorub'a into English.
  arXiv  Detail & Related papers  (2021-03-07T01:24:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.