Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition

• URL: http://arxiv.org/abs/2010.03192v1
• Date: Wed, 7 Oct 2020 05:58:28 GMT
• Title: Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition
• Authors: Anshuman Tripathi, Jaeyoung Kim, Qian Zhang, Han Lu, Hasim Sak
• Abstract summary: We present a Transformer-Transducer model architecture and a training technique to unify streaming and non-streaming speech recognition models into one model. We show that we can run this model in a Y-model architecture with the top layers running in parallel in low latency and high latency modes. This allows us to have streaming speech recognition results with limited latency and delayed speech recognition results with large improvements in accuracy.
• Score: 16.082949461807335
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper we present a Transformer-Transducer model architecture and a training technique to unify streaming and non-streaming speech recognition models into one model. The model is composed of a stack of transformer layers for audio encoding with no lookahead or right context and an additional stack of transformer layers on top trained with variable right context. In inference time, the context length for the variable context layers can be changed to trade off the latency and the accuracy of the model. We also show that we can run this model in a Y-model architecture with the top layers running in parallel in low latency and high latency modes. This allows us to have streaming speech recognition results with limited latency and delayed speech recognition results with large improvements in accuracy (20% relative improvement for voice-search task). We show that with limited right context (1-2 seconds of audio) and small additional latency (50-100 milliseconds) at the end of decoding, we can achieve similar accuracy with models using unlimited audio right context. We also present optimizations for audio and label encoders to speed up the inference in streaming and non-streaming speech decoding.

Related papers

• Autoregressive Diffusion Transformer for Text-to-Speech Synthesis [39.32761051774537]
  We propose encoding audio as vector sequences in continuous space $mathbb Rd$ and autoregressively generating these sequences. High-bitrate continuous speech representation enables almost flawless reconstruction, allowing our model to achieve nearly perfect speech editing.
  arXiv  Detail & Related papers  (2024-06-08T18:57:13Z)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [51.70360630470263]
  Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video. We propose Frieren, a V2A model based on rectified flow matching. Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment.
  arXiv  Detail & Related papers  (2024-06-01T06:40:22Z)
• FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis [77.06890315052563]
  We propose FastLTS, a non-autoregressive end-to-end model which can directly synthesize high-quality speech audios from unconstrained talking videos with low latency. Experiments show that our model achieves $19.76times$ speedup for audio generation compared with the current autoregressive model on input sequences of 3 seconds.
  arXiv  Detail & Related papers  (2022-07-08T10:10:39Z)
• On Comparison of Encoders for Attention based End to End Speech Recognition in Standalone and Rescoring Mode [1.7704011486040847]
  Non-streaming models provide better performance as they look at the entire audio context. We show that the Transformer model offers acceptable WER with the lowest latency requirements. We highlight the importance of CNN front-end with Transformer architecture to achieve comparable word error rates (WER)
  arXiv  Detail & Related papers  (2022-06-26T09:12:27Z)
• SoundStream: An End-to-End Neural Audio Codec [78.94923131038682]
  We present SoundStream, a novel neural audio system that can efficiently compress speech, music and general audio. SoundStream relies on a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. We are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency.
  arXiv  Detail & Related papers  (2021-07-07T15:45:42Z)
• Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition [19.971343876930767]
  We present a novel two-pass approach to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model. Our model adopts the hybrid CTC/attention architecture, in which the conformer layers in the encoder are modified. Experiments on the open 170-hour AISHELL-1 dataset show that, the proposed method can unify the streaming and non-streaming model simply and efficiently.
  arXiv  Detail & Related papers  (2020-12-10T06:54:54Z)
• Streaming Simultaneous Speech Translation with Augmented Memory Transformer [29.248366441276662]
  Transformer-based models have achieved state-of-the-art performance on speech translation tasks. We propose an end-to-end transformer-based sequence-to-sequence model, equipped with an augmented memory transformer encoder.
  arXiv  Detail & Related papers  (2020-10-30T18:28:42Z)
• VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition [60.462770498366524]
  We introduce VoiceFilter-Lite, a single-channel source separation model that runs on the device to preserve only the speech signals from a target user. We show that such a model can be quantized as a 8-bit integer model and run in realtime.
  arXiv  Detail & Related papers  (2020-09-09T14:26:56Z)
• Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition [8.046120977786702]
  Transformer has achieved competitive performance against state-of-the-art end-to-end models in automatic speech recognition (ASR) The original Transformer, with encoder-decoder architecture, is only suitable for offline ASR. We show that this architecture, named Conv-Transformer Transducer, achieves competitive performance on LibriSpeech dataset (3.6% WER on test-clean) without external language models.
  arXiv  Detail & Related papers  (2020-08-13T08:20:02Z)
• Relative Positional Encoding for Speech Recognition and Direct Translation [72.64499573561922]
  We adapt the relative position encoding scheme to the Speech Transformer. As a result, the network can better adapt to the variable distributions present in speech data.
  arXiv  Detail & Related papers  (2020-05-20T09:53:06Z)
• Streaming automatic speech recognition with the transformer model [59.58318952000571]
  We propose a transformer based end-to-end ASR system for streaming ASR. We apply time-restricted self-attention for the encoder and triggered attention for the encoder-decoder attention mechanism. Our proposed streaming transformer architecture achieves 2.8% and 7.2% WER for the "clean" and "other" test data of LibriSpeech.
  arXiv  Detail & Related papers  (2020-01-08T18:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.