NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation

• URL: http://arxiv.org/abs/2606.13121v1
• Date: Thu, 11 Jun 2026 09:49:46 GMT
• Title: NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation
• Abstract summary: Simultaneous speech-to-speech translation aims to enable near-real-time communication by minimizing latency.<n>However, the excessive pursuit of low latency often results in fragmented chunk-wise speech.<n>We introduce a fluency-aware optimization framework designed to discover the sweet spot between the low-latency benefits of simultaneous translation and the natural flow of consecutive translation.
• Score: 47.343134511282834
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Simultaneous speech-to-speech translation aims to enable near-real-time communication by minimizing latency, offering a compelling, real-time alternative to the high latency of consecutive translation. However, the excessive pursuit of low latency often results in fragmented chunk-wise speech. Consequently, listeners are subjected to an unnatural acoustic flow punctuated by frequent pauses, which could increase their cognitive load. To bridge this gap, we introduce a fluency-aware optimization framework designed to discover the sweet spot between the low-latency benefits of simultaneous translation and the natural flow of consecutive translation. Our framework minimizes inter-chunk silences by leveraging model-internal signals, including linguistic diversity and induced temporal variability in speech durations. Experiments on short- and long-form benchmarks show that our framework produces natural speech flow while maintaining competitive latency and translation quality.

Related papers

• Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-Like Strategies [6.010207559477024]
  Simultaneous Machine Translation (SiMT) requires high-quality translations under strict real-time constraints.<n>We extend the action space of SiMT with four adaptive actions: Sentence_Cut, Drop, Partial_Summarization and Pronominalization.<n>We adapt these actions in a large language model (LLM) framework and construct training references through action-aware prompting.
  arXiv  Detail & Related papers  (2026-01-16T05:26:16Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• Context-aware TFL: A Universal Context-aware Contrastive Learning Framework for Temporal Forgery Localization [60.73623588349311]
  We propose a universal context-aware contrastive learning framework (UniCaCLF) for temporal forgery localization.<n>Our approach leverages supervised contrastive learning to discover and identify forged instants by means of anomaly detection.<n>An efficient context-aware contrastive coding is introduced to further push the limit of instant feature distinguishability between genuine and forged instants.
  arXiv  Detail & Related papers  (2025-06-10T06:40:43Z)
• Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing [15.134076873312809]
  Cross-lingual dubbing system translates speech from one language to another while preserving key characteristics such as duration, speaker identity, and speaking speed.<n>We propose a discrete diffusion-based speech-to-unit translation model with explicit duration control, enabling time-aligned translation.<n>We then synthesize speech based on the predicted units and source identity with a conditional flow matching model.
  arXiv  Detail & Related papers  (2025-05-27T08:43:28Z)
• High-Fidelity Simultaneous Speech-To-Speech Translation [75.69884829562591]
  We introduce Hibiki, a decoder-only model for simultaneous speech translation.<n>Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation.
  arXiv  Detail & Related papers  (2025-02-05T17:18:55Z)
• What does it take to get state of the art in simultaneous speech-to-speech translation? [0.0]
  We study the latency characteristics observed in simultaneous speech-to-speech model's performance. We propose methods to minimize latency spikes and improve overall performance.
  arXiv  Detail & Related papers  (2024-09-02T06:04:07Z)
• A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation [48.84039953531355]
  We propose a novel non-autoregressive generation framework for simultaneous speech translation (NAST-S2X) NAST-S2X integrates speech-to-text and speech-to-speech tasks into a unified end-to-end framework. It achieves high-quality simultaneous interpretation within a delay of less than 3 seconds and provides a 28 times decoding speedup in offline generation.
  arXiv  Detail & Related papers  (2024-06-11T04:25:48Z)
• Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff [49.75167556773752]
  Blockwise self-attentional encoder models have emerged as one promising end-to-end approach to simultaneous speech translation. We propose a modified incremental blockwise beam search incorporating local agreement or hold-$n$ policies for quality-latency control.
  arXiv  Detail & Related papers  (2023-09-20T14:59:06Z)
• Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training [40.71155396456831]
  Simultaneous speech-to-speech translation is widely useful but extremely challenging. It needs to generate target-language speech concurrently with the source-language speech, with only a few seconds delay. Current approaches accumulate latencies progressively when the speaker talks faster, and introduce unnatural pauses when the speaker talks slower. We propose Self-Adaptive Translation (SAT) which flexibly adjusts the length of translations to accommodate different source speech rates.
  arXiv  Detail & Related papers  (2020-10-20T06:02:15Z)
• Low Latency ASR for Simultaneous Speech Translation [27.213294097841853]
  We have worked on several techniques for reducing the latency for both components, the automatic speech recognition and the speech translation module. We combined run-on decoding with a technique for identifying stable partial hypotheses when stream decoding and a protocol for dynamic output update. This combination reduces the latency at word level, where the words are final and will never be updated again in the future, from 18.1s to 1.1s without sacrificing performance.
  arXiv  Detail & Related papers  (2020-03-22T13:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.