REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation

• URL: http://arxiv.org/abs/2508.04946v1
• Date: Thu, 07 Aug 2025 00:25:58 GMT
• Title: REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation
• Authors: Nameer Hirschkind, Joseph Liu, Mahesh Kumar Nandwana, Xiao Yu,
• Abstract summary: Simultaneous Speech Translation (SimulST) systems stream in audio while simultaneously emitting translated text or speech.<n>We introduce a strategy to optimize this tradeoff: wait for more input only if you gain information by doing so.<n>We present Regularized Entropy INformation Adaptation (REINA), a novel loss to train an adaptive policy using an existing non-streaming translation model.
• Score: 3.230443390004258
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Simultaneous Speech Translation (SimulST) systems stream in audio while simultaneously emitting translated text or speech. Such systems face the significant challenge of balancing translation quality and latency. We introduce a strategy to optimize this tradeoff: wait for more input only if you gain information by doing so. Based on this strategy, we present Regularized Entropy INformation Adaptation (REINA), a novel loss to train an adaptive policy using an existing non-streaming translation model. We derive REINA from information theory principles and show that REINA helps push the reported Pareto frontier of the latency/quality tradeoff over prior works. Utilizing REINA, we train a SimulST model on French, Spanish and German, both from and into English. Training on only open source or synthetically generated data, we achieve state-of-the-art (SOTA) streaming results for models of comparable size. We also introduce a metric for streaming efficiency, quantitatively showing REINA improves the latency/quality trade-off by as much as 21% compared to prior approaches, normalized against non-streaming baseline BLEU scores.

Related papers

• Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization [0.19791587637442667]
  Inverse Text Normalization (ITN) is crucial for converting spoken Automatic Speech Recognition (ASR) outputs into well-formatted written text.<n>We introduce a streaming pretrained language model for ITN, leveraging pretrained linguistic representations for improved robustness.<n>Our method achieves accuracy comparable to non-streaming ITN and surpasses existing streaming ITN models on a Vietnamese dataset.
  arXiv  Detail & Related papers  (2025-05-30T05:41:03Z)
• SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models [64.40250409933752]
  We build upon our previous publication by implementing a simple and efficient non-autoregressive (NAR) TTS framework, termed SimpleSpeech 2. SimpleSpeech 2 effectively combines the strengths of both autoregressive (AR) and non-autoregressive (NAR) methods. We show a significant improvement in generation performance and generation speed compared to our previous work and other state-of-the-art (SOTA) large-scale TTS models.
  arXiv  Detail & Related papers  (2024-08-25T17:07:39Z)
• FASST: Fast LLM-based Simultaneous Speech Translation [9.65638081954595]
  Simultaneous speech translation (SST) takes streaming speech input and generates text translation on the fly. We propose FASST, a fast large language model based method for streaming speech translation. Experiment results show that FASST achieves the best quality-latency trade-off.
  arXiv  Detail & Related papers  (2024-08-18T10:12:39Z)
• DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation [29.76274107159478]
  Non-autoregressive Transformers (NATs) are applied in direct speech-to-speech translation systems. We introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. Our strategies result in a notable improvement of about +7 ASR-BLEU for English-Spanish (En-Es) and +2 ASR-BLEU for English-French (En-Fr) on the CVSS benchmark.
  arXiv  Detail & Related papers  (2024-05-22T01:10:39Z)
• Data-Driven Adaptive Simultaneous Machine Translation [51.01779863078624]
  We propose a novel and efficient training scheme for adaptive SimulMT. Our method outperforms all strong baselines in terms of translation quality and latency.
  arXiv  Detail & Related papers  (2022-04-27T02:40:21Z)
• Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation [75.86581380817464]
  A SimulST system generally includes two components: the pre-decision that aggregates the speech information and the policy that decides to read or write. This paper proposes to model the adaptive policy by adapting the Continuous Integrate-and-Fire (CIF) Compared with monotonic multihead attention (MMA), our method has the advantage of simpler computation, superior quality at low latency, and better generalization to long utterances.
  arXiv  Detail & Related papers  (2022-03-22T23:33:18Z)
• Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation [49.916963624249355]
  A UNMT model is trained on the pseudo parallel data with translated source, and natural source sentences in inference. The source discrepancy between training and inference hinders the translation performance of UNMT models. We propose an online self-training approach, which simultaneously uses the pseudo parallel data natural source, translated target to mimic the inference scenario.
  arXiv  Detail & Related papers  (2022-03-16T04:50:27Z)
• Anticipation-free Training for Simultaneous Translation [70.85761141178597]
  Simultaneous translation (SimulMT) speeds up the translation process by starting to translate before the source sentence is completely available. Existing methods increase latency or introduce adaptive read-write policies for SimulMT models to handle local reordering and improve translation quality. We propose a new framework that decomposes the translation process into the monotonic translation step and the reordering step.
  arXiv  Detail & Related papers  (2022-01-30T16:29:37Z)
• The USYD-JD Speech Translation System for IWSLT 2021 [85.64797317290349]
  This paper describes the University of Sydney& JD's joint submission of the IWSLT 2021 low resource speech translation task. We trained our models with the officially provided ASR and MT datasets. To achieve better translation performance, we explored the most recent effective strategies, including back translation, knowledge distillation, multi-feature reranking and transductive finetuning.
  arXiv  Detail & Related papers  (2021-07-24T09:53:34Z)
• A Technical Report: BUT Speech Translation Systems [2.9327503320877457]
  The paper describes the BUT's speech translation systems. The systems are English$longrightarrow$German offline speech translation systems. A large degradation is observed when translating ASR hypothesis compared to the oracle input text.
  arXiv  Detail & Related papers  (2020-10-22T10:52:31Z)
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation [53.8171136907856]
  We introduce a set of simple yet effective data augmentation strategies dubbed cutoff. cutoff relies on sampling consistency and thus adds little computational overhead. cutoff consistently outperforms adversarial training and achieves state-of-the-art results on the IWSLT2014 German-English dataset.
  arXiv  Detail & Related papers  (2020-09-29T07:08:35Z)
• Re-translation versus Streaming for Simultaneous Translation [14.800214853561823]
  We study a problem in which revisions to the hypothesis beyond strictly appending words are permitted. In this setting, we compare custom streaming approaches to re-translation. We find re-translation to be as good or better than state-of-the-art streaming systems.
  arXiv  Detail & Related papers  (2020-04-07T18:27:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.