An Automatic Quality Metric for Evaluating Simultaneous Interpretation

• URL: http://arxiv.org/abs/2407.06650v2
• Date: Fri, 13 Sep 2024 14:39:33 GMT
• Title: An Automatic Quality Metric for Evaluating Simultaneous Interpretation
• Authors: Mana Makinae, Katsuhito Sudoh, Mararu Yamada, Satoshi Nakamura,
• Abstract summary: Simultaneous interpretation (SI) starts translation before the original speech has finished. We propose an automatic evaluation metric for SI and simultaneous machine translation (SiMT) focusing on word order synchronization.
• Score: 13.009481258370702
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Simultaneous interpretation (SI), the translation of one language to another in real time, starts translation before the original speech has finished. Its evaluation needs to consider both latency and quality. This trade-off is challenging especially for distant word order language pairs such as English and Japanese. To handle this word order gap, interpreters maintain the word order of the source language as much as possible to keep up with original language to minimize its latency while maintaining its quality, whereas in translation reordering happens to keep fluency in the target language. This means outputs synchronized with the source language are desirable based on the real SI situation, and it's a key for further progress in computational SI and simultaneous machine translation (SiMT). In this work, we propose an automatic evaluation metric for SI and SiMT focusing on word order synchronization. Our evaluation metric is based on rank correlation coefficients, leveraging cross-lingual pre-trained language models. Our experimental results on NAIST-SIC-Aligned and JNPC showed our metrics' effectiveness to measure word order synchronization between source and target language.

Related papers

• BiVert: Bidirectional Vocabulary Evaluation using Relations for Machine Translation [4.651581292181871]
  We propose a bidirectional semantic-based evaluation method designed to assess the sense distance of the translation from the source text. This approach employs the comprehensive multilingual encyclopedic dictionary BabelNet. Factual analysis shows a strong correlation between the average evaluation scores generated by our method and the human assessments across various machine translation systems for English-German language pair.
  arXiv  Detail & Related papers  (2024-03-06T08:02:21Z)
• Language Model is a Branch Predictor for Simultaneous Machine Translation [73.82754138171587]
  We propose incorporating branch prediction techniques in SiMT tasks to reduce translation latency. We utilize a language model as a branch predictor to predict potential branch directions. When the actual source word deviates from the predicted source word, we use the real source word to decode the output again, replacing the predicted output.
  arXiv  Detail & Related papers  (2023-12-22T07:32:47Z)
• Motion2Language, unsupervised learning of synchronized semantic motion segmentation [0.0]
  We investigate building a sequence to sequence architecture for motion to language translation and synchronization. The aim is to translate motion capture inputs into English natural-language descriptions, such that the descriptions are generated synchronously with the actions performed. We propose a new recurrent formulation of local attention that is suited for synchronous/live text generation, as well as an improved motion encoder architecture.
  arXiv  Detail & Related papers  (2023-10-16T17:16:32Z)
• Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters [71.02335065794384]
  We introduce target factors in a transformer model to predict durations jointly with target language phoneme sequences. We show that our model improves translation quality and isochrony compared to previous work.
  arXiv  Detail & Related papers  (2023-05-22T16:36:04Z)
• Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation [37.48971774827332]
  We propose to generate ST tokens out-of-order while remembering how to re-order them later. We examine two variants of such operation sequences which enable generation of monotonic transcriptions and non-monotonic translations.
  arXiv  Detail & Related papers  (2022-11-11T02:29:28Z)
• Code-Switching without Switching: Language Agnostic End-to-End Speech Translation [68.8204255655161]
  We treat speech recognition and translation as one unified end-to-end speech translation problem. By training LAST with both input languages, we decode speech into one target language, regardless of the input language.
  arXiv  Detail & Related papers  (2022-10-04T10:34:25Z)
• Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement [57.72846454929923]
  We create a benchmark dataset, emphHJQE, where the expert translators directly annotate poorly translated words. We propose two tag correcting strategies, namely tag refinement strategy and tree-based annotation strategy, to make the TER-based artificial QE corpus closer to emphHJQE. The results show our proposed dataset is more consistent with human judgement and also confirm the effectiveness of the proposed tag correcting strategies.
  arXiv  Detail & Related papers  (2022-09-13T02:37:12Z)
• SimulEval: An Evaluation Toolkit for Simultaneous Translation [59.02724214432792]
  Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario. SimulEval is an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation.
  arXiv  Detail & Related papers  (2020-07-31T17:44:41Z)
• Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing [11.564158965143418]
  We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser. We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task. Our method is simple and intuitive, and does not require human judgements for training.
  arXiv  Detail & Related papers  (2020-04-30T03:32:34Z)
• Self-Attention with Cross-Lingual Position Representation [112.05807284056337]
  Position encoding (PE) is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences. Due to word order divergences in different languages, modeling the cross-lingual positional relationships might help SANs tackle this problem. We augment SANs with emphcross-lingual position representations to model the bilingually aware latent structure for the input sentence.
  arXiv  Detail & Related papers  (2020-04-28T05:23:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.