Stream-level Latency Evaluation for Simultaneous Machine Translation

• URL: http://arxiv.org/abs/2104.08817v1
• Date: Sun, 18 Apr 2021 11:16:17 GMT
• Title: Stream-level Latency Evaluation for Simultaneous Machine Translation
• Authors: Javier Iranzo-S\'anchez and Jorge Civera and Alfons Juan
• Abstract summary: Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. This work proposes a stream-level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation.
• Score: 5.50178437495268
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. Simultaneous translation systems need to find a trade-off between translation quality and response time, and with this purpose multiple latency measures have been proposed. However, latency evaluations for simultaneous translation are estimated at the sentence level, not taking into account the sequential nature of a streaming scenario. Indeed, these sentence-level latency measures are not well suited for continuous stream translation resulting in figures that are not coherent with the simultaneous translation policy of the system being assessed. This work proposes a stream-level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation, that is successfully evaluated on streaming conditions for a reference IWSLT task.

Related papers

• CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation [17.473263201972483]
  Simultaneous speech translation (SimulST) systems must balance translation quality with response time. There has been a longstanding belief that current metrics yield unrealistically high latency measurements in unsegmented streaming settings.
  arXiv  Detail & Related papers  (2024-10-21T13:42:19Z)
• Average Token Delay: A Duration-aware Latency Metric for Simultaneous Translation [16.954965417930254]
  We propose a novel latency evaluation metric for simultaneous translation called emphAverage Token Delay (ATD) We demonstrate its effectiveness through analyses simulating user-side latency based on Ear-Voice Span (EVS)
  arXiv  Detail & Related papers  (2023-11-24T08:53:52Z)
• Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff [49.75167556773752]
  Blockwise self-attentional encoder models have emerged as one promising end-to-end approach to simultaneous speech translation. We propose a modified incremental blockwise beam search incorporating local agreement or hold-$n$ policies for quality-latency control.
  arXiv  Detail & Related papers  (2023-09-20T14:59:06Z)
• End-to-End Evaluation for Low-Latency Simultaneous Speech Translation [55.525125193856084]
  We propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions. This includes the segmentation of the audio as well as the run-time of the different components. We also compare different approaches to low-latency speech translation using this framework.
  arXiv  Detail & Related papers  (2023-08-07T09:06:20Z)
• Average Token Delay: A Latency Metric for Simultaneous Translation [21.142539715996673]
  We propose a novel latency evaluation metric called Average Token Delay (ATD) We discuss the advantage of ATD using simulated examples and also investigate the differences between ATD and Average Lagging with simultaneous translation experiments.
  arXiv  Detail & Related papers  (2022-11-22T06:45:13Z)
• Data-Driven Adaptive Simultaneous Machine Translation [51.01779863078624]
  We propose a novel and efficient training scheme for adaptive SimulMT. Our method outperforms all strong baselines in terms of translation quality and latency.
  arXiv  Detail & Related papers  (2022-04-27T02:40:21Z)
• From Simultaneous to Streaming Machine Translation by Leveraging Streaming History [4.831134508326648]
  Simultaneous Machine Translation is the task of incrementally translating an input sentence before it is fully available. Streaming MT can be understood as an extension of Simultaneous MT to the incremental translation of a continuous input text stream. In this work, a state-of-the-art simultaneous sentence-level MT system is extended to the streaming setup by leveraging the streaming history.
  arXiv  Detail & Related papers  (2022-03-04T17:41:45Z)
• Anticipation-free Training for Simultaneous Translation [70.85761141178597]
  Simultaneous translation (SimulMT) speeds up the translation process by starting to translate before the source sentence is completely available. Existing methods increase latency or introduce adaptive read-write policies for SimulMT models to handle local reordering and improve translation quality. We propose a new framework that decomposes the translation process into the monotonic translation step and the reordering step.
  arXiv  Detail & Related papers  (2022-01-30T16:29:37Z)
• Presenting Simultaneous Translation in Limited Space [0.0]
  Some methods of automatic simultaneous translation of a long-form speech allow revisions of outputs, trading accuracy for low latency. Subtitling must be shown promptly, incrementally, and with adequate time for reading. We propose a way how to estimate the overall usability of the combination of automatic translation and subtitling by measuring the quality, latency, and stability on a test set.
  arXiv  Detail & Related papers  (2020-09-18T18:37:03Z)
• SimulEval: An Evaluation Toolkit for Simultaneous Translation [59.02724214432792]
  Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario. SimulEval is an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation.
  arXiv  Detail & Related papers  (2020-07-31T17:44:41Z)
• Towards Streaming Perception [70.68520310095155]
  We present an approach that coherently integrates latency and accuracy into a single metric for real-time online perception. The key insight behind this metric is to jointly evaluate the output of the entire perception stack at every time instant. We focus on the illustrative tasks of object detection and instance segmentation in urban video streams, and contribute a novel dataset with high-quality and temporally-dense annotations.
  arXiv  Detail & Related papers  (2020-05-21T01:51:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.