SimulEval: An Evaluation Toolkit for Simultaneous Translation
- URL: http://arxiv.org/abs/2007.16193v1
- Date: Fri, 31 Jul 2020 17:44:41 GMT
- Title: SimulEval: An Evaluation Toolkit for Simultaneous Translation
- Authors: Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, Juan Pino
- Abstract summary: Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario.
SimulEval is an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation.
- Score: 59.02724214432792
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Simultaneous translation on both text and speech focuses on a real-time and
low-latency scenario where the model starts translating before reading the
complete source input. Evaluating simultaneous translation models is more
complex than offline models because the latency is another factor to consider
in addition to translation quality. The research community, despite its growing
focus on novel modeling approaches to simultaneous translation, currently lacks
a universal evaluation procedure. Therefore, we present SimulEval, an
easy-to-use and general evaluation toolkit for both simultaneous text and
speech translation. A server-client scheme is introduced to create a
simultaneous translation scenario, where the server sends source input and
receives predictions for evaluation and the client executes customized
policies. Given a policy, it automatically performs simultaneous decoding and
collectively reports several popular latency metrics. We also adapt latency
metrics from text simultaneous translation to the speech task. Additionally,
SimulEval is equipped with a visualization interface to provide better
understanding of the simultaneous decoding process of a system. SimulEval has
already been extensively used for the IWSLT 2020 shared task on simultaneous
speech translation. Code will be released upon publication.
Related papers
- An Automatic Quality Metric for Evaluating Simultaneous Interpretation [13.009481258370702]
Simultaneous interpretation (SI) starts translation before the original speech has finished.
We propose an automatic evaluation metric for SI and simultaneous machine translation (SiMT) focusing on word order synchronization.
arXiv Detail & Related papers (2024-07-09T08:21:40Z) - Leveraging Timestamp Information for Serialized Joint Streaming
Recognition and Translation [51.399695200838586]
We propose a streaming Transformer-Transducer (T-T) model able to jointly produce many-to-one and one-to-many transcription and translation using a single decoder.
Experiments on it,es,de->en prove the effectiveness of our approach, enabling the generation of one-to-many joint outputs with a single decoder for the first time.
arXiv Detail & Related papers (2023-10-23T11:00:27Z) - End-to-End Evaluation for Low-Latency Simultaneous Speech Translation [55.525125193856084]
We propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions.
This includes the segmentation of the audio as well as the run-time of the different components.
We also compare different approaches to low-latency speech translation using this framework.
arXiv Detail & Related papers (2023-08-07T09:06:20Z) - Modeling Sequential Sentence Relation to Improve Cross-lingual Dense
Retrieval [87.11836738011007]
We propose a multilingual multilingual language model called masked sentence model (MSM)
MSM consists of a sentence encoder to generate the sentence representations, and a document encoder applied to a sequence of sentence vectors from a document.
To train the model, we propose a masked sentence prediction task, which masks and predicts the sentence vector via a hierarchical contrastive loss with sampled negatives.
arXiv Detail & Related papers (2023-02-03T09:54:27Z) - Bilingual Synchronization: Restoring Translational Relationships with
Editing Operations [2.0411082897313984]
We consider a more general setting which assumes an initial target sequence, that must be transformed into a valid translation of the source.
Our results suggest that one single generic edit-based system, once fine-tuned, can compare with, or even outperform, dedicated systems specifically trained for these tasks.
arXiv Detail & Related papers (2022-10-24T12:25:44Z) - Anticipation-free Training for Simultaneous Translation [70.85761141178597]
Simultaneous translation (SimulMT) speeds up the translation process by starting to translate before the source sentence is completely available.
Existing methods increase latency or introduce adaptive read-write policies for SimulMT models to handle local reordering and improve translation quality.
We propose a new framework that decomposes the translation process into the monotonic translation step and the reordering step.
arXiv Detail & Related papers (2022-01-30T16:29:37Z) - SimulSLT: End-to-End Simultaneous Sign Language Translation [55.54237194555432]
Existing sign language translation methods need to read all the videos before starting the translation.
We propose SimulSLT, the first end-to-end simultaneous sign language translation model.
SimulSLT achieves BLEU scores that exceed the latest end-to-end non-simultaneous sign language translation model.
arXiv Detail & Related papers (2021-12-08T11:04:52Z) - RealTranS: End-to-End Simultaneous Speech Translation with Convolutional
Weighted-Shrinking Transformer [33.876412404781846]
RealTranS is an end-to-end model for simultaneous speech translation.
It maps speech features into text space with a weighted-shrinking operation and a semantic encoder.
Experiments show that RealTranS with the Wait-K-Stride-N strategy outperforms prior end-to-end models.
arXiv Detail & Related papers (2021-06-09T06:35:46Z) - SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End
Simultaneous Speech Translation [23.685648804345984]
Simultaneous text translation and end-to-end speech translation have recently made great progress but little work has combined these tasks together.
We investigate how to adapt simultaneous text translation methods such as wait-k and monotonic multihead attention to end-to-end simultaneous speech translation by introducing a pre-decision module.
A detailed analysis is provided on the latency-quality trade-offs of combining fixed and flexible pre-decision with fixed and flexible policies.
arXiv Detail & Related papers (2020-11-03T22:47:58Z) - Streaming Simultaneous Speech Translation with Augmented Memory
Transformer [29.248366441276662]
Transformer-based models have achieved state-of-the-art performance on speech translation tasks.
We propose an end-to-end transformer-based sequence-to-sequence model, equipped with an augmented memory transformer encoder.
arXiv Detail & Related papers (2020-10-30T18:28:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.