A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation

• URL: http://arxiv.org/abs/2606.15059v1
• Date: Sat, 13 Jun 2026 02:20:49 GMT
• Title: A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation
• Abstract summary: Simultaneous speech-to-speech translation (SimulS2ST) enables real-time cross-lingual communication.<n>We present a practical evaluation method for long-form SimulS2ST.
• Score: 9.807543197382016
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simultaneous speech-to-speech translation (SimulS2ST) enables real-time cross-lingual communication, but existing evaluation has focused largely on short or pre-segmented speech rather than long-form, continuous input. Prior approaches are difficult to reproduce and make assumptions that do not hold for end-to-end systems. We present a practical evaluation method for long-form SimulS2ST. Given source speech, pre-segmented source transcripts, and reference translations, we run automatic speech recognition (ASR) and forced alignment on the generated target speech to recover token-level timestamps, then apply a sentence-embedding-based aligner to match the target text to its corresponding source sentences. This enables sentence-level computation of latency and quality metrics, including YAAL and xCOMET, which are then aggregated into final system-level scores. Experiments on representative SimulS2ST systems show that the method is effective in practice and reveal that current systems suffer from substantial latency accumulation on long speech.

Related papers

• Few-Shot Contrastive Adaptation for Audio Abuse Detection in Low-Resource Indic Languages [3.5238606794194816]
  Abusive speech detection is becoming increasingly important as social media shifts towards voice-based interaction.<n>We investigate whether Contrastive Language-Audio Pre-training can support abusive speech detection directly from audio.
  arXiv  Detail & Related papers  (2026-04-10T08:23:03Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model [76.06585781346601]
  Speech language models (Speech LMs) enable end-to-end speech-text modelling within a single model.<n>The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality.
  arXiv  Detail & Related papers  (2025-06-04T23:53:49Z)
• S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models [14.060679420379516]
  End-to-end speech large language models ((LLMs)) extend the capabilities of text-based models to directly process and generate audio tokens.<n>This often leads to a decline in reasoning and generation performance compared to text input.<n>We propose S2SBench, a benchmark designed to quantify performance degradation in Speech LLMs.
  arXiv  Detail & Related papers  (2025-05-20T14:42:20Z)
• OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching [3.05024318465243]
  OZSpeech is the first TTS method to explore optimal transport conditional flow matching with one-step sampling.<n>Our approach operates on disentangled, factorized components of speech in token format, enabling accurate modeling of each speech attribute.<n> Experimental results show that our method achieves promising performance over existing methods in content accuracy, naturalness, prosody generation, and speaker style preservation.
  arXiv  Detail & Related papers  (2025-05-19T07:31:55Z)
• SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation [14.57248739077317]
  This paper proposes SimulS2S-LLM, which trains speech LLMs offline and employs a test-time policy to guide simultaneous inference.<n>SimulS2S-LLM achieves simultaneous speech-to-speech translation (Simul-S2ST) by predicting discrete output speech tokens and then synthesising output speech using a pre-trained vocoder.
  arXiv  Detail & Related papers  (2025-04-22T01:05:32Z)
• Long-Form Speech Generation with Spoken Language Models [64.29591880693468]
  Textless spoken language models struggle to generate plausible speech past tens of seconds.<n>We derive SpeechSSM, the first speech language model family to learn from and sample long-form spoken audio.<n>SpeechSSMs leverage recent advances in linear-time sequence modeling to greatly surpass current Transformer spoken LMs in coherence and efficiency.
  arXiv  Detail & Related papers  (2024-12-24T18:56:46Z)
• Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems [55.99999020778169]
  We study a function that can predict the forthcoming words and estimate the time remaining until the end of an utterance. We develop a cross-attention-based algorithm that incorporates both acoustic and linguistic information. Results demonstrate the proposed model's ability to predict upcoming words and estimate future EOU events up to 300ms prior to the actual EOU.
  arXiv  Detail & Related papers  (2024-09-30T06:29:58Z)
• StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning [48.84039953531356]
  StreamSpeech is a direct Simul-S2ST model that jointly learns translation and simultaneous policy. Experiments on CVSS benchmark demonstrate that StreamSpeech achieves state-of-the-art performance in both offline S2ST and Simul-S2ST tasks.
  arXiv  Detail & Related papers  (2024-06-05T08:24:22Z)
• DiariST: Streaming Speech Translation with Speaker Diarization [53.595990270899414]
  We propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech.
  arXiv  Detail & Related papers  (2023-09-14T19:33:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.