Promptformer: Prompted Conformer Transducer for ASR

• URL: http://arxiv.org/abs/2401.07360v1
• Date: Sun, 14 Jan 2024 20:14:35 GMT
• Title: Promptformer: Prompted Conformer Transducer for ASR
• Authors: Sergio Duarte-Torres, Arunasish Sen, Aman Rana, Lukas Drude, Alejandro Gomez-Alanis, Andreas Schwarz, Leif R\"adel, Volker Leutnant
• Abstract summary: We introduce a novel mechanism inspired by hyper-prompting to fuse textual context with acoustic representations in the attention mechanism. Results on a test set with multi-turn interactions show that our method achieves 5.9% relative word error rate reduction (rWERR) over a strong baseline.
• Score: 40.88399609719793
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Context cues carry information which can improve multi-turn interactions in automatic speech recognition (ASR) systems. In this paper, we introduce a novel mechanism inspired by hyper-prompting to fuse textual context with acoustic representations in the attention mechanism. Results on a test set with multi-turn interactions show that our method achieves 5.9% relative word error rate reduction (rWERR) over a strong baseline. We show that our method does not degrade in the absence of context and leads to improvements even if the model is trained without context. We further show that leveraging a pre-trained sentence-piece model for context embedding generation can outperform an external BERT model.

Related papers

• Towards interfacing large language models with ASR systems using confidence measures and prompting [54.39667883394458]
  This work investigates post-hoc correction of ASR transcripts with large language models (LLMs) To avoid introducing errors into likely accurate transcripts, we propose a range of confidence-based filtering methods. Our results indicate that this can improve the performance of less competitive ASR systems.
  arXiv  Detail & Related papers  (2024-07-31T08:00:41Z)
• Quantifying the Role of Textual Predictability in Automatic Speech Recognition [13.306122574236232]
  A long-standing question in automatic speech recognition research is how to attribute errors to the ability of a model to model the acoustics. We validate a novel approach which models error rates as a function of relative textual predictability. We show how this approach can be used straightforwardly in diagnosing and improving ASR.
  arXiv  Detail & Related papers  (2024-07-23T14:47:25Z)
• Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring [4.819085609772069]
  We propose a novel approach for enhancing contextual recognition within ASR systems via semantic lattice processing. Our solution consists of using Hidden Markov Models and Gaussian Mixture Models (HMM-GMM) along with Deep Neural Networks (DNN) models for better accuracy. We demonstrate the effectiveness of our proposed framework on the LibriSpeech dataset with empirical analyses.
  arXiv  Detail & Related papers  (2023-10-14T23:16:05Z)
• Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations [59.10748929158525]
  Abstract Representations (AMR) can greatly improve the performance of unsupervised syntactically controlled paraphrase generation. Our proposed model, AMR-enhanced Paraphrase Generator (AMRPG), encodes the AMR graph and the constituency parses the input sentence into two disentangled semantic and syntactic embeddings. Experiments show that AMRPG generates more accurate syntactically controlled paraphrases, both quantitatively and qualitatively, compared to the existing unsupervised approaches.
  arXiv  Detail & Related papers  (2022-11-02T04:58:38Z)
• Contextual-Utterance Training for Automatic Speech Recognition [65.4571135368178]
  We propose a contextual-utterance training technique which makes use of the previous and future contextual utterances. Also, we propose a dual-mode contextual-utterance training technique for streaming automatic speech recognition (ASR) systems. The proposed technique is able to reduce both the WER and the average last token emission latency by more than 6% and 40ms relative.
  arXiv  Detail & Related papers  (2022-10-27T08:10:44Z)
• DialAug: Mixing up Dialogue Contexts in Contrastive Learning for Robust Conversational Modeling [3.3578533367912025]
  We propose a framework that incorporates augmented versions of a dialogue context into the learning objective. We show that our proposed augmentation method outperforms previous data augmentation approaches.
  arXiv  Detail & Related papers  (2022-04-15T23:39:41Z)
• A Light-weight contextual spelling correction model for customizing transducer-based speech recognition systems [42.05399301143457]
  We introduce a light-weight contextual spelling correction model to correct context-related recognition errors. Experiments show that the model improves baseline ASR model performance with about 50% relative word error rate reduction. The model also shows excellent performance for out-of-vocabulary terms not seen during training.
  arXiv  Detail & Related papers  (2021-08-17T08:14:37Z)
• Weak-Attention Suppression For Transformer Based Speech Recognition [33.30436927415777]
  We propose Weak-Attention Suppression (WAS), a method that dynamically induces sparsity in attention probabilities. We demonstrate that WAS leads to consistent Word Error Rate (WER) improvement over strong transformer baselines.
  arXiv  Detail & Related papers  (2020-05-18T23:49:40Z)
• A Controllable Model of Grounded Response Generation [122.7121624884747]
  Current end-to-end neural conversation models inherently lack the flexibility to impose semantic control in the response generation process. We propose a framework that we call controllable grounded response generation (CGRG) We show that using this framework, a transformer based model with a novel inductive attention mechanism, trained on a conversation-like Reddit dataset, outperforms strong generation baselines.
  arXiv  Detail & Related papers  (2020-05-01T21:22:08Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.