Average Token Delay: A Duration-aware Latency Metric for Simultaneous Translation

• URL: http://arxiv.org/abs/2311.14353v2
• Date: Mon, 27 Nov 2023 16:55:29 GMT
• Title: Average Token Delay: A Duration-aware Latency Metric for Simultaneous Translation
• Authors: Yasumasa Kano, Katsuhito Sudoh, Satoshi Nakamura
• Abstract summary: We propose a novel latency evaluation metric for simultaneous translation called emphAverage Token Delay (ATD) We demonstrate its effectiveness through analyses simulating user-side latency based on Ear-Voice Span (EVS)
• Score: 16.954965417930254
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Simultaneous translation is a task in which the translation begins before the end of an input speech segment. Its evaluation should be conducted based on latency in addition to quality, and for users, the smallest possible amount of latency is preferable. Most existing metrics measure latency based on the start timings of partial translations and ignore their duration. This means such metrics do not penalize the latency caused by long translation output, which delays the comprehension of users and subsequent translations. In this work, we propose a novel latency evaluation metric for simultaneous translation called \emph{Average Token Delay} (ATD) that focuses on the duration of partial translations. We demonstrate its effectiveness through analyses simulating user-side latency based on Ear-Voice Span (EVS). In our experiment, ATD had the highest correlation with EVS among baseline latency metrics under most conditions.

Related papers

• CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation [17.473263201972483]
  Simultaneous speech translation (SimulST) systems must balance translation quality with response time. There has been a longstanding belief that current metrics yield unrealistically high latency measurements in unsegmented streaming settings.
  arXiv  Detail & Related papers  (2024-10-21T13:42:19Z)
• Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling [73.5602474095954]
  We study the non-asymptotic performance of approximation schemes with delayed updates under Markovian sampling. Our theoretical findings shed light on the finite-time effects of delays for a broad class of algorithms.
  arXiv  Detail & Related papers  (2024-02-19T03:08:02Z)
• Language Model is a Branch Predictor for Simultaneous Machine Translation [73.82754138171587]
  We propose incorporating branch prediction techniques in SiMT tasks to reduce translation latency. We utilize a language model as a branch predictor to predict potential branch directions. When the actual source word deviates from the predicted source word, we use the real source word to decode the output again, replacing the predicted output.
  arXiv  Detail & Related papers  (2023-12-22T07:32:47Z)
• Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff [49.75167556773752]
  Blockwise self-attentional encoder models have emerged as one promising end-to-end approach to simultaneous speech translation. We propose a modified incremental blockwise beam search incorporating local agreement or hold-$n$ policies for quality-latency control.
  arXiv  Detail & Related papers  (2023-09-20T14:59:06Z)
• Average Token Delay: A Latency Metric for Simultaneous Translation [21.142539715996673]
  We propose a novel latency evaluation metric called Average Token Delay (ATD) We discuss the advantage of ATD using simulated examples and also investigate the differences between ATD and Average Lagging with simultaneous translation experiments.
  arXiv  Detail & Related papers  (2022-11-22T06:45:13Z)
• Data-Driven Adaptive Simultaneous Machine Translation [51.01779863078624]
  We propose a novel and efficient training scheme for adaptive SimulMT. Our method outperforms all strong baselines in terms of translation quality and latency.
  arXiv  Detail & Related papers  (2022-04-27T02:40:21Z)
• Anticipation-free Training for Simultaneous Translation [70.85761141178597]
  Simultaneous translation (SimulMT) speeds up the translation process by starting to translate before the source sentence is completely available. Existing methods increase latency or introduce adaptive read-write policies for SimulMT models to handle local reordering and improve translation quality. We propose a new framework that decomposes the translation process into the monotonic translation step and the reordering step.
  arXiv  Detail & Related papers  (2022-01-30T16:29:37Z)
• Stream-level Latency Evaluation for Simultaneous Machine Translation [5.50178437495268]
  Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. This work proposes a stream-level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation.
  arXiv  Detail & Related papers  (2021-04-18T11:16:17Z)
• Presenting Simultaneous Translation in Limited Space [0.0]
  Some methods of automatic simultaneous translation of a long-form speech allow revisions of outputs, trading accuracy for low latency. Subtitling must be shown promptly, incrementally, and with adequate time for reading. We propose a way how to estimate the overall usability of the combination of automatic translation and subtitling by measuring the quality, latency, and stability on a test set.
  arXiv  Detail & Related papers  (2020-09-18T18:37:03Z)
• SimulEval: An Evaluation Toolkit for Simultaneous Translation [59.02724214432792]
  Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario. SimulEval is an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation.
  arXiv  Detail & Related papers  (2020-07-31T17:44:41Z)
• Low Latency ASR for Simultaneous Speech Translation [27.213294097841853]
  We have worked on several techniques for reducing the latency for both components, the automatic speech recognition and the speech translation module. We combined run-on decoding with a technique for identifying stable partial hypotheses when stream decoding and a protocol for dynamic output update. This combination reduces the latency at word level, where the words are final and will never be updated again in the future, from 18.1s to 1.1s without sacrificing performance.
  arXiv  Detail & Related papers  (2020-03-22T13:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.