Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation

• URL: http://arxiv.org/abs/2406.04541v1
• Date: Thu, 6 Jun 2024 22:39:43 GMT
• Title: Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation
• Authors: Keqi Deng, Philip C. Woodland,
• Abstract summary: This paper presents the LS-Transducer-SST, a label-synchronous neural transducer for simultaneous speech translation (SST) The LS-Transducer-SST dynamically decides when to emit translation tokens based on an Auto-regressive Integrate-and-Fire mechanism. Experiments on the Fisher-CallHome Spanish (Es-En) and MuST-C En-De data show that the LS-Transducer-SST gives a better quality-latency trade-off than existing popular methods.
• Score: 14.410024368174872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While the neural transducer is popular for online speech recognition, simultaneous speech translation (SST) requires both streaming and re-ordering capabilities. This paper presents the LS-Transducer-SST, a label-synchronous neural transducer for SST, which naturally possesses these two properties. The LS-Transducer-SST dynamically decides when to emit translation tokens based on an Auto-regressive Integrate-and-Fire (AIF) mechanism. A latency-controllable AIF is also proposed, which can control the quality-latency trade-off either only during decoding, or it can be used in both decoding and training. The LS-Transducer-SST can naturally utilise monolingual text-only data via its prediction network which helps alleviate the key issue of data sparsity for E2E SST. During decoding, a chunk-based incremental joint decoding technique is designed to refine and expand the search space. Experiments on the Fisher-CallHome Spanish (Es-En) and MuST-C En-De data show that the LS-Transducer-SST gives a better quality-latency trade-off than existing popular methods. For example, the LS-Transducer-SST gives a 3.1/2.9 point BLEU increase (Es-En/En-De) relative to CAAT at a similar latency and a 1.4 s reduction in average lagging latency with similar BLEU scores relative to Wait-k.

Related papers

• DiariST: Streaming Speech Translation with Speaker Diarization [53.595990270899414]
  We propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech.
  arXiv  Detail & Related papers  (2023-09-14T19:33:27Z)
• Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments [49.38965743465124]
  This paper introduces a streaming Transformer-Transducer that jointly generates automatic speech recognition (ASR) and speech translation (ST) outputs using a single decoder. Experiments in monolingual and multilingual settings demonstrate that our approach achieves the best quality-latency balance.
  arXiv  Detail & Related papers  (2023-07-07T02:26:18Z)
• UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units [64.61596752343837]
  We present a novel two-pass direct S2ST architecture, UnitY, which first generates textual representations and predicts discrete acoustic units. We enhance the model performance by subword prediction in the first-pass decoder. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass.
  arXiv  Detail & Related papers  (2022-12-15T18:58:28Z)
• LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers [71.76680102779765]
  Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. We propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers.
  arXiv  Detail & Related papers  (2022-11-05T04:03:55Z)
• Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation [10.650573361117669]
  Semi-supervised training and synthetic code-switched data can improve the bilingual ASR system on code-switching speech. Our final system achieves 25% mixed error rate (MER) on the ASCEND English/Mandarin code-switching test set.
  arXiv  Detail & Related papers  (2022-10-21T19:42:41Z)
• Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation [35.31787938396058]
  This paper takes the first step on streaming spoken language understanding (SLU) and speech translation (ST) using a blockwise streaming Transformer. We propose a cross-lingual encoding method, which employs a CTC branch optimized with target language translations. Experimental results show that the blockwise streaming Transformer achieves competitive results compared to offline models.
  arXiv  Detail & Related papers  (2022-04-19T14:38:40Z)
• Large-Scale Streaming End-to-End Speech Translation with Neural Transducers [35.2855796745394]
  We introduce a streaming end-to-end speech translation (ST) model to convert audio signals to texts in other languages directly. Compared with cascaded ST that performs ASR followed by text-based machine translation (MT), the proposed Transformer transducer (TT)-based ST model drastically reduces inference latency. We extend TT-based ST to multilingual ST, which generates texts of multiple languages at the same time.
  arXiv  Detail & Related papers  (2022-04-11T18:18:53Z)
• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation [76.13334392868208]
  Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues. In this work, we explore self-supervised pre-training with unlabeled speech data and data augmentation to tackle this issue.
  arXiv  Detail & Related papers  (2022-04-06T17:59:22Z)
• Incremental Speech Synthesis For Speech-To-Speech Translation [23.951060578077445]
  We focus on improving the incremental synthesis performance of TTS models. With a simple data augmentation strategy based on prefixes, we are able to improve the incremental TTS quality to approach offline performance. We propose latency metrics tailored to S2ST applications, and investigate methods for latency reduction in this context.
  arXiv  Detail & Related papers  (2021-10-15T17:20:28Z)
• Adaptive Feature Selection for End-to-End Speech Translation [87.07211937607102]
  We propose adaptive feature selection (AFS) for encoder-decoder based E2E speech translation (ST) We first pre-train an ASR encoder and apply AFS to dynamically estimate the importance of each encoded speech feature to SR. We take L0DROP as the backbone for AFS, and adapt it to sparsify speech features with respect to both temporal and feature dimensions.
  arXiv  Detail & Related papers  (2020-10-16T17:21:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.