PromptASR for contextualized ASR with controllable style

• URL: http://arxiv.org/abs/2309.07414v3
• Date: Wed, 24 Jan 2024 08:29:09 GMT
• Title: PromptASR for contextualized ASR with controllable style
• Authors: Xiaoyu Yang, Wei Kang, Zengwei Yao, Yifan Yang, Liyong Guo, Fangjun Kuang, Long Lin, Daniel Povey
• Abstract summary: We propose PromptASR, a framework that integrates prompts in end-to-end automatic speech recognition systems. The proposed system achieves 21.9% and 6.8% relative word error rate reductions on a book reading dataset. An additional style prompt can be given to the text encoder and guide the ASR system to output different styles of transcriptions.
• Score: 19.493184060647728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompts are crucial to large language models as they provide context information such as topic or logical relationships. Inspired by this, we propose PromptASR, a framework that integrates prompts in end-to-end automatic speech recognition (E2E ASR) systems to achieve contextualized ASR with controllable style of transcriptions. Specifically, a dedicated text encoder encodes the text prompts and the encodings are injected into the speech encoder by cross-attending the features from two modalities. When using the ground truth text from preceding utterances as content prompt, the proposed system achieves 21.9% and 6.8% relative word error rate reductions on a book reading dataset and an in-house dataset compared to a baseline ASR system. The system can also take word-level biasing lists as prompt to improve recognition accuracy on rare words. An additional style prompt can be given to the text encoder and guide the ASR system to output different styles of transcriptions. The code is available at icefall.

Related papers

• Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM [48.71951982716363]
  Text-to-speech (TTS) models have been widely adopted to enhance automatic speech recognition (ASR) systems. We propose Hard- Synth, a novel ASR data augmentation method that leverages large language models (LLMs) and advanced zero-shot TTS. Our approach employs LLMs to generate diverse in-domain text through rewriting, without relying on additional text data.
  arXiv  Detail & Related papers  (2024-11-20T09:49:37Z)
• Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Words [10.2138250640885]
  We develop a large language model (LLM) based automatic speech recognition (ASR) system that can be contextualized by providing keywords in text prompts. We adopt decoder-only architecture and use our in-house LLM, PLaMo-100B, pre-trained from scratch using datasets dominated by Japanese and English texts as the decoder.
  arXiv  Detail & Related papers  (2024-08-15T08:50:58Z)
• Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation [67.89838237013078]
  Named entity recognition (NER) models often struggle with noisy inputs. We propose a more realistic setting in which only noisy text and its NER labels are available. We employ a multi-view training framework that improves robust NER without retrieving text during inference.
  arXiv  Detail & Related papers  (2024-07-26T07:30:41Z)
• An efficient text augmentation approach for contextualized Mandarin speech recognition [4.600045052545344]
  Our study proposes to leverage extensive text-only datasets and contextualize pre-trained ASR models. To contextualize a pre-trained CIF-based ASR, we construct a codebook using limited speech-text data. Our experiments on diverse Mandarin test sets demonstrate that our TA approach significantly boosts recognition performance.
  arXiv  Detail & Related papers  (2024-06-14T11:53:14Z)
• Instruction-Guided Scene Text Recognition [51.853730414264625]
  We propose a novel instruction-guided scene text recognition (IGTR) paradigm that formulates STR as an instruction learning problem. We develop lightweight instruction encoder, cross-modal feature fusion module and multi-task answer head, which guides nuanced text image understanding. IGTR outperforms existing models by significant margins, while maintaining a small model size and efficient inference speed.
  arXiv  Detail & Related papers  (2024-01-31T14:13:01Z)
• Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation [27.926862030684926]
  We introduce a novel conversational ASR system, extending the Conformer encoder-decoder model with cross-modal conversational representation. Our approach combines pre-trained speech and text models through a specialized encoder and a modal-level mask input. By introducing both cross-modal and conversational representations into the decoder, our model retains context over longer sentences without information loss.
  arXiv  Detail & Related papers  (2023-10-22T11:57:33Z)
• Investigating the Sensitivity of Automatic Speech Recognition Systems to Phonetic Variation in L2 Englishes [3.198144010381572]
  This work demonstrates a method of probing an ASR system to discover how it handles phonetic variation across a number of L2 Englishes. It is demonstrated that the behaviour of the ASR is systematic and consistent across speakers with similar spoken varieties.
  arXiv  Detail & Related papers  (2023-05-12T11:29:13Z)
• Deliberation Model for On-Device Spoken Language Understanding [69.5587671262691]
  We propose a novel deliberation-based approach to end-to-end (E2E) spoken language understanding (SLU) We show that our approach can significantly reduce the degradation when moving from natural speech to synthetic speech training.
  arXiv  Detail & Related papers  (2022-04-04T23:48:01Z)
• Label-Synchronous Speech-to-Text Alignment for ASR Using Forward and Backward Transformers [49.403414751667135]
  This paper proposes a novel label-synchronous speech-to-text alignment technique for automatic speech recognition (ASR) The proposed method re-defines the speech-to-text alignment as a label-synchronous text mapping problem. Experiments using the corpus of spontaneous Japanese (CSJ) demonstrate that the proposed method provides an accurate utterance-wise alignment.
  arXiv  Detail & Related papers  (2021-04-21T03:05:12Z)
• On the Impact of Word Error Rate on Acoustic-Linguistic Speech Emotion Recognition: An Update for the Deep Learning Era [0.0]
  We create transcripts from the original speech by applying three modern ASR systems. For extraction and learning of acoustic speech features, we utilise openSMILE, openXBoW, DeepSpectrum, and auDeep. We achieve state-of-the-art unweighted average recall values of $73.6,%$ and $73.8,%$ on the speaker-independent development and test partitions of IEMOCAP.
  arXiv  Detail & Related papers  (2021-04-20T17:10:01Z)
• Improving Readability for Automatic Speech Recognition Transcription [50.86019112545596]
  We propose a novel NLP task called ASR post-processing for readability (APR) APR aims to transform the noisy ASR output into a readable text for humans and downstream tasks while maintaining the semantic meaning of the speaker. We compare fine-tuned models based on several open-sourced and adapted pre-trained models with the traditional pipeline method.
  arXiv  Detail & Related papers  (2020-04-09T09:26:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.