Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems

• URL: http://arxiv.org/abs/2512.17648v1
• Date: Fri, 19 Dec 2025 14:48:59 GMT
• Title: Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems
• Abstract summary: Streaming Speech-to-Text Translation (StreamST) requires producing translations concurrently with incoming speech.<n>SimulEval repository is no longer maintained and does not support systems that revise their outputs.<n>We introduce simulstream, the first open-source framework dedicated to unified evaluation and demonstration of StreamST systems.
• Score: 32.41110835446445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Streaming Speech-to-Text Translation (StreamST) requires producing translations concurrently with incoming speech, imposing strict latency constraints and demanding models that balance partial-information decision-making with high translation quality. Research efforts on the topic have so far relied on the SimulEval repository, which is no longer maintained and does not support systems that revise their outputs. In addition, it has been designed for simulating the processing of short segments, rather than long-form audio streams, and it does not provide an easy method to showcase systems in a demo. As a solution, we introduce simulstream, the first open-source framework dedicated to unified evaluation and demonstration of StreamST systems. Designed for long-form speech processing, it supports not only incremental decoding approaches, but also re-translation methods, enabling for their comparison within the same framework both in terms of quality and latency. In addition, it also offers an interactive web interface to demo any system built within the tool.

Related papers

• Streaming Speech-to-Text Translation with a SpeechLLM [21.176799517305096]
  SpeechLLM promises to exploit paralinguistic information in the speech and to reduce cascaded errors.<n>Existing SpeechLLM systems are slow since they wait for a complete utterance of audio before outputting a translation.<n>This work proposes an LLM-based architecture for real streaming speech-to-text translation.<n>In experiments on different language pairs, the system achieves a translation quality close to the non-streaming baseline, but with a latency of only 1-2 seconds.
  arXiv  Detail & Related papers  (2026-05-14T12:32:57Z)
• Voxtral Realtime [134.66962524291424]
  Voxtral Realtime is a streaming automatic speech recognition model.<n>It matches offline transcription quality at sub-second latency.<n>We release the model weights under the Apache 2.0 license.
  arXiv  Detail & Related papers  (2026-02-11T19:17:10Z)
• Asynchronous Pipeline Parallelism for Real-Time Multilingual Lip Synchronization in Video Communication Systems [0.21748200848556343]
  The proposed architecture integrates translation, speech processing, and lip-synchronization modules within a pipeline-parallel design.<n>It reduces end-to-end latency by up to 3.1 times compared to sequential approaches.<n>This work advances the development of low-latency, resource-efficient multimodal communication frameworks for next-generation AIoT systems.
  arXiv  Detail & Related papers  (2025-12-20T11:23:18Z)
• StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model [20.978001644716063]
  Streaming speech translation (StreamST) requires determining appropriate timing, known as policy, to generate translations.<n>Existing StreamST methods typically operate on sentence-level speech segments, referred to as simultaneous speech translation (SimulST)<n>We propose StreamUni, which achieves StreamST through a unified Large Speech-Language Model (LSLM)
  arXiv  Detail & Related papers  (2025-07-10T14:28:39Z)
• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling [50.537794606598254]
  StreamMel is a pioneering single-stage streaming TTS framework that models continuous mel-spectrograms.<n>It enables low-latency, autoregressive synthesis while preserving high speaker similarity and naturalness.<n>It even achieves performance comparable to offline systems while supporting efficient real-time generation.
  arXiv  Detail & Related papers  (2025-06-14T16:53:39Z)
• Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling [76.23539797803681]
  Existing methods primarily use a look mechanism, relying on future text to achieve natural streaming speech synthesis.<n>We propose LE, a streaming framework for generating high-quality speech frame-by-frame.<n> Experimental results suggest that the LE outperforms current streaming TTS methods and achieves comparable performance over sentence-level TTS systems.
  arXiv  Detail & Related papers  (2025-05-26T08:25:01Z)
• High-Fidelity Simultaneous Speech-To-Speech Translation [75.69884829562591]
  We introduce Hibiki, a decoder-only model for simultaneous speech translation.<n>Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation.
  arXiv  Detail & Related papers  (2025-02-05T17:18:55Z)
• DiariST: Streaming Speech Translation with Speaker Diarization [53.595990270899414]
  We propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech.
  arXiv  Detail & Related papers  (2023-09-14T19:33:27Z)
• UniST: Unified End-to-end Model for Streaming and Non-streaming Speech Translation [12.63410397982031]
  We develop a unified model (UniST) which supports streaming and non-streaming speech translation. Experiments on the most popular speech-to-text translation benchmark dataset, MuST-C, show that UniST achieves significant improvement for non-streaming ST.
  arXiv  Detail & Related papers  (2021-09-15T15:22:10Z)
• A low latency ASR-free end to end spoken language understanding system [11.413018142161249]
  This work proposes a system that has a small enough footprint to run on small micro-controllers and embedded systems with minimal latency. Given a streaming input speech signal, the proposed system can process it segment-by-segment without the need to have the entire stream at the moment of processing. Experiments show that the proposed system yields state-of-the-art performance with the advantage of low latency and a much smaller model when compared to other published works on the same task.
  arXiv  Detail & Related papers  (2020-11-10T04:16:56Z)
• Streaming Simultaneous Speech Translation with Augmented Memory Transformer [29.248366441276662]
  Transformer-based models have achieved state-of-the-art performance on speech translation tasks. We propose an end-to-end transformer-based sequence-to-sequence model, equipped with an augmented memory transformer encoder.
  arXiv  Detail & Related papers  (2020-10-30T18:28:42Z)
• SimulEval: An Evaluation Toolkit for Simultaneous Translation [59.02724214432792]
  Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario. SimulEval is an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation.
  arXiv  Detail & Related papers  (2020-07-31T17:44:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.