Deliberation of Streaming RNN-Transducer by Non-autoregressive Decoding

• URL: http://arxiv.org/abs/2112.11442v1
• Date: Wed, 1 Dec 2021 01:34:28 GMT
• Title: Deliberation of Streaming RNN-Transducer by Non-autoregressive Decoding
• Authors: Weiran Wang, Ke Hu, Tara Sainath
• Abstract summary: The method performs a few refinement steps, where each step shares a transformer decoder that attends to both text features and audio features. We show that, conditioned on hypothesis alignments of a streaming RNN-T model, our method obtains significantly more accurate recognition results than the first-pass RNN-T.
• Score: 21.978994865937786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose to deliberate the hypothesis alignment of a streaming RNN-T model with the previously proposed Align-Refine non-autoregressive decoding method and its improved versions. The method performs a few refinement steps, where each step shares a transformer decoder that attends to both text features (extracted from alignments) and audio features, and outputs complete updated alignments. The transformer decoder is trained with the CTC loss which facilitates parallel greedy decoding, and performs full-context attention to capture label dependencies. We improve Align-Refine by introducing cascaded encoder that captures more audio context before refinement, and alignment augmentation which enforces learning label dependency. We show that, conditioned on hypothesis alignments of a streaming RNN-T model, our method obtains significantly more accurate recognition results than the first-pass RNN-T, with only small amount of model parameters.

Related papers

• Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition [20.052245837954175]
  We propose an efficient and accurate streaming speech recognition model based on the FastConformer architecture. We introduce an activation caching mechanism to enable the non-autoregressive encoder to operate autoregressively during inference. A hybrid CTC/RNNT architecture which utilizes a shared encoder with both a CTC and RNNT decoder to boost the accuracy and save computation.
  arXiv  Detail & Related papers  (2023-12-27T21:04:26Z)
• Learned layered coding for Successive Refinement in the Wyner-Ziv Problem [18.134147308944446]
  We propose a data-driven approach to explicitly learn the progressive encoding of a continuous source. This setup refers to the successive refinement of the Wyner-Ziv coding problem. We demonstrate that RNNs can explicitly retrieve layered binning solutions akin to scalable nested quantization.
  arXiv  Detail & Related papers  (2023-11-06T12:45:32Z)
• AICT: An Adaptive Image Compression Transformer [18.05997169440533]
  We propose a more straightforward yet effective Tranformer-based channel-wise auto-regressive prior model, resulting in an absolute image compression transformer (ICT) The proposed ICT can capture both global and local contexts from the latent representations. We leverage a learnable scaling module with a sandwich ConvNeXt-based pre/post-processor to accurately extract more compact latent representation.
  arXiv  Detail & Related papers  (2023-07-12T11:32:02Z)
• Streaming Audio-Visual Speech Recognition with Alignment Regularization [69.30185151873707]
  We propose a streaming AV-ASR system based on a hybrid connectionist temporal classification ( CTC)/attention neural network architecture. The proposed AV-ASR model achieves WERs of 2.0% and 2.6% on the Lip Reading Sentences 3 dataset in an offline and online setup.
  arXiv  Detail & Related papers  (2022-11-03T20:20:47Z)
• Streaming Align-Refine for Non-autoregressive Deliberation [42.748839817396046]
  We propose a streaming non-autoregressive (non-AR) decoding algorithm to deliberate the hypothesis alignment of a streaming RNN-T model. Our algorithm facilitates a simple greedy decoding procedure, and at the same time is capable of producing the decoding result at each frame with limited right context. Experiments on voice search datasets and Librispeech show that with reasonable right context, our streaming model performs as well as the offline counterpart.
  arXiv  Detail & Related papers  (2022-04-15T17:24:39Z)
• Error Correction Code Transformer [92.10654749898927]
  We propose to extend for the first time the Transformer architecture to the soft decoding of linear codes at arbitrary block lengths. We encode each channel's output dimension to high dimension for better representation of the bits information to be processed separately. The proposed approach demonstrates the extreme power and flexibility of Transformers and outperforms existing state-of-the-art neural decoders by large margins at a fraction of their time complexity.
  arXiv  Detail & Related papers  (2022-03-27T15:25:58Z)
• Sequence Transduction with Graph-based Supervision [96.04967815520193]
  We present a new transducer objective function that generalizes the RNN-T loss to accept a graph representation of the labels. We demonstrate that transducer-based ASR with CTC-like lattice achieves better results compared to standard RNN-T.
  arXiv  Detail & Related papers  (2021-11-01T21:51:42Z)
• On Addressing Practical Challenges for RNN-Transduce [72.72132048437751]
  We adapt a well-trained RNN-T model to a new domain without collecting the audio data. We obtain word-level confidence scores by utilizing several types of features calculated during decoding. The proposed time stamping method can get less than 50ms word timing difference on average.
  arXiv  Detail & Related papers  (2021-04-27T23:31:43Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)
• Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss [14.755108017449295]
  We present an end-to-end speech recognition model with Transformer encoders that can be used in a streaming speech recognition system. Transformer computation blocks based on self-attention are used to encode both audio and label sequences independently. We present results on the LibriSpeech dataset showing that limiting the left context for self-attention makes decoding computationally tractable for streaming.
  arXiv  Detail & Related papers  (2020-02-07T00:04:04Z)
• Streaming automatic speech recognition with the transformer model [59.58318952000571]
  We propose a transformer based end-to-end ASR system for streaming ASR. We apply time-restricted self-attention for the encoder and triggered attention for the encoder-decoder attention mechanism. Our proposed streaming transformer architecture achieves 2.8% and 7.2% WER for the "clean" and "other" test data of LibriSpeech.
  arXiv  Detail & Related papers  (2020-01-08T18:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.