Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translation

• URL: http://arxiv.org/abs/2604.09916v1
• Date: Fri, 10 Apr 2026 21:20:55 GMT
• Title: Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translation
• Abstract summary: Simultaneous Speech Translation (SimulST) requires balancing high translation quality with low latency.<n>Recent work introduced REINA, a method that trains a Read/Write policy based on estimating the information gain of reading more audio.<n>We find that information-based policies often lack temporal context, leading the policy to bias itself toward reading most of the audio before starting to write.
• Score: 6.531488967831291
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Simultaneous Speech Translation (SimulST) requires balancing high translation quality with low latency. Recent work introduced REINA, a method that trains a Read/Write policy based on estimating the information gain of reading more audio. However, we find that information-based policies often lack temporal context, leading the policy to bias itself toward reading most of the audio before starting to write. We improve REINA using two distinct strategies: a supervised alignment network (REINA-SAN) and a timestep-augmented network (REINA-TAN). Our results demonstrate that while both methods significantly outperform the baseline and resolve stability issues, REINA-TAN provides a slightly superior Pareto frontier for streaming efficiency, whereas REINA-SAN offers more robustness against 'read loops'. Applied to Whisper, both methods improve the pareto frontier of streaming efficiency as measured by Normalized Streaming Efficiency (NoSE) scores up to 7.1% over existing competitive baselines.

Related papers

• Coverage Improvement and Fast Convergence of On-policy Preference Learning [67.36750525893514]
  Online on-policy preference learning algorithms for language model alignment can significantly outperform their offline counterparts.<n>We analyze how the sampling policy's coverage evolves throughout on-policy training.<n>We develop principled on-policy schemes for reward distillation in the general function class setting.
  arXiv  Detail & Related papers  (2026-01-13T10:46:06Z)
• A Task-Oriented Evaluation Framework for Text Normalization in Modern NLP Pipelines [0.0]
  This study proposes a novel, task-oriented approach to evaluate stemming methods.<n>It considers three aspects: (1) the utility of stemming using Stemming Effectiveness Score (SES), (2) the impact of stemming on downstream tasks using Model Performance Delta (MPD), and (3) the semantic similarity between stemmed and original words using Average Normalized Levenshtein Distance (ANLD)<n>Our study provides a valuable tool for distinguishing between potential efficiency gains (high SES) and meaning preservation (low ANLD)
  arXiv  Detail & Related papers  (2025-11-25T15:35:42Z)
• Overcoming Latency Bottlenecks in On-Device Speech Translation: A Cascaded Approach with Alignment-Based Streaming MT [19.133273093370896]
  This paper tackles several challenges when integrating Automatic Speech Recognition (ASR) and Machine Translation (MT) for real-time, on-device streaming speech translation.<n>We propose a simultaneous translation approach that effectively balances translation quality and latency.<n>We apply our approach to an on-device bilingual conversational speech translation and demonstrate that our techniques outperform baselines in terms of latency and quality.
  arXiv  Detail & Related papers  (2025-08-18T21:00:11Z)
• REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation [3.230443390004258]
  Simultaneous Speech Translation (SimulST) systems stream in audio while simultaneously emitting translated text or speech.<n>We introduce a strategy to optimize this tradeoff: wait for more input only if you gain information by doing so.<n>We present Regularized Entropy INformation Adaptation (REINA), a novel loss to train an adaptive policy using an existing non-streaming translation model.
  arXiv  Detail & Related papers  (2025-08-07T00:25:58Z)
• DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation [29.76274107159478]
  Non-autoregressive Transformers (NATs) are applied in direct speech-to-speech translation systems. We introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. Our strategies result in a notable improvement of about +7 ASR-BLEU for English-Spanish (En-Es) and +2 ASR-BLEU for English-French (En-Fr) on the CVSS benchmark.
  arXiv  Detail & Related papers  (2024-05-22T01:10:39Z)
• Data-Driven Adaptive Simultaneous Machine Translation [51.01779863078624]
  We propose a novel and efficient training scheme for adaptive SimulMT. Our method outperforms all strong baselines in terms of translation quality and latency.
  arXiv  Detail & Related papers  (2022-04-27T02:40:21Z)
• End-to-End Active Speaker Detection [58.7097258722291]
  We propose an end-to-end training network where feature learning and contextual predictions are jointly learned. We also introduce intertemporal graph neural network (iGNN) blocks, which split the message passing according to the main sources of context in the ASD problem. Experiments show that the aggregated features from the iGNN blocks are more suitable for ASD, resulting in state-of-the art performance.
  arXiv  Detail & Related papers  (2022-03-27T08:55:28Z)
• Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation [75.86581380817464]
  A SimulST system generally includes two components: the pre-decision that aggregates the speech information and the policy that decides to read or write. This paper proposes to model the adaptive policy by adapting the Continuous Integrate-and-Fire (CIF) Compared with monotonic multihead attention (MMA), our method has the advantage of simpler computation, superior quality at low latency, and better generalization to long utterances.
  arXiv  Detail & Related papers  (2022-03-22T23:33:18Z)
• Time-domain Speech Enhancement with Generative Adversarial Learning [53.74228907273269]
  This paper proposes a new framework called Time-domain Speech Enhancement Generative Adversarial Network (TSEGAN) TSEGAN is an extension of the generative adversarial network (GAN) in time-domain with metric evaluation to mitigate the scaling problem. In addition, we provide a new method based on objective function mapping for the theoretical analysis of the performance of Metric GAN.
  arXiv  Detail & Related papers  (2021-03-30T08:09:49Z)
• Improving noise robust automatic speech recognition with single-channel time-domain enhancement network [100.1041336974175]
  We show that a single-channel time-domain denoising approach can significantly improve ASR performance. We show that single-channel noise reduction can still improve ASR performance.
  arXiv  Detail & Related papers  (2020-03-09T09:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.