Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism

• URL: http://arxiv.org/abs/2207.00883v1
• Date: Sat, 2 Jul 2022 17:17:47 GMT
• Title: Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism
• Authors: Kun Wei, Pengcheng Guo, Ning Jiang
• Abstract summary: We propose to explicitly model the inter-sentential information in a Transformer based end-to-end architecture for conversational speech recognition. We show the effectiveness of our proposed method on several open-source dialogue corpora and the proposed method consistently improved the performance from the utterance-level Transformer-based ASR models.
• Score: 20.782319059183173
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based models have demonstrated their effectiveness in automatic speech recognition (ASR) tasks and even shown superior performance over the conventional hybrid framework. The main idea of Transformers is to capture the long-range global context within an utterance by self-attention layers. However, for scenarios like conversational speech, such utterance-level modeling will neglect contextual dependencies that span across utterances. In this paper, we propose to explicitly model the inter-sentential information in a Transformer based end-to-end architecture for conversational speech recognition. Specifically, for the encoder network, we capture the contexts of previous speech and incorporate such historic information into current input by a context-aware residual attention mechanism. For the decoder, the prediction of current utterance is also conditioned on the historic linguistic information through a conditional decoder framework. We show the effectiveness of our proposed method on several open-source dialogue corpora and the proposed method consistently improved the performance from the utterance-level Transformer-based ASR models.

Related papers

• SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation [56.913182262166316]
  Chain-of-Information Generation (CoIG) is a method for decoupling semantic and perceptual information in large-scale speech generation. SpeechGPT-Gen is efficient in semantic and perceptual information modeling. It markedly excels in zero-shot text-to-speech, zero-shot voice conversion, and speech-to-speech dialogue.
  arXiv  Detail & Related papers  (2024-01-24T15:25:01Z)
• Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring [4.819085609772069]
  We propose a novel approach for enhancing contextual recognition within ASR systems via semantic lattice processing. Our solution consists of using Hidden Markov Models and Gaussian Mixture Models (HMM-GMM) along with Deep Neural Networks (DNN) models for better accuracy. We demonstrate the effectiveness of our proposed framework on the LibriSpeech dataset with empirical analyses.
  arXiv  Detail & Related papers  (2023-10-14T23:16:05Z)
• Channel-aware Decoupling Network for Multi-turn Dialogue Comprehension [81.47133615169203]
  We propose compositional learning for holistic interaction across utterances beyond the sequential contextualization from PrLMs. We employ domain-adaptive training strategies to help the model adapt to the dialogue domains. Experimental results show that our method substantially boosts the strong PrLM baselines in four public benchmark datasets.
  arXiv  Detail & Related papers  (2023-01-10T13:18:25Z)
• Contextual-Utterance Training for Automatic Speech Recognition [65.4571135368178]
  We propose a contextual-utterance training technique which makes use of the previous and future contextual utterances. Also, we propose a dual-mode contextual-utterance training technique for streaming automatic speech recognition (ASR) systems. The proposed technique is able to reduce both the WER and the average last token emission latency by more than 6% and 40ms relative.
  arXiv  Detail & Related papers  (2022-10-27T08:10:44Z)
• Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASR [25.75615870266786]
  We propose an audio-textual cross-modal representation extractor to learn contextual representations directly from preceding speech. The effectiveness of the proposed approach is validated on several Mandarin conversation corpora.
  arXiv  Detail & Related papers  (2022-07-03T13:32:24Z)
• Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification [66.62686601948455]
  We exploit the scope of the transformer distillation method that is specifically designed for knowledge distillation from a transformer based language model to a transformer based speech model. We achieve an intent classification accuracy of 99.10% and 88.79% for Fluent speech corpus and ATIS database, respectively.
  arXiv  Detail & Related papers  (2021-08-05T13:08:13Z)
• Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation [28.51624095262708]
  We present a novel large-context end-to-end automatic speech recognition (E2E-ASR) model and its effective training method based on knowledge distillation. This paper proposes a hierarchical transformer-based large-context E2E-ASR model that combines the transformer architecture with hierarchical encoder-decoder based large-context modeling.
  arXiv  Detail & Related papers  (2021-02-16T03:15:15Z)
• Relative Positional Encoding for Speech Recognition and Direct Translation [72.64499573561922]
  We adapt the relative position encoding scheme to the Speech Transformer. As a result, the network can better adapt to the variable distributions present in speech data.
  arXiv  Detail & Related papers  (2020-05-20T09:53:06Z)
• End-to-end speech-to-dialog-act recognition [38.58540444573232]
  We present an end-to-end model which directly converts speech into dialog acts without the deterministic transcription process. In the proposed model, the dialog act recognition network is conjunct with an acoustic-to-word ASR model at its latent layer. The entire network is fine-tuned in an end-to-end manner.
  arXiv  Detail & Related papers  (2020-04-23T18:44:27Z)
• Streaming automatic speech recognition with the transformer model [59.58318952000571]
  We propose a transformer based end-to-end ASR system for streaming ASR. We apply time-restricted self-attention for the encoder and triggered attention for the encoder-decoder attention mechanism. Our proposed streaming transformer architecture achieves 2.8% and 7.2% WER for the "clean" and "other" test data of LibriSpeech.
  arXiv  Detail & Related papers  (2020-01-08T18:58:02Z)
• Transformer-based language modeling and decoding for conversational speech recognition [0.0]
  We focus on decoding efficiently in a weighted finite-state transducer framework. We showcase an approach to lattice re-scoring that allows for longer range history captured by a transfomer-based language model.
  arXiv  Detail & Related papers  (2020-01-04T23:27:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.