Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI

• URL: http://arxiv.org/abs/2501.06129v1
• Date: Fri, 10 Jan 2025 17:35:06 GMT
• Title: Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI
• Authors: Yuya Asano, Sabit Hassan, Paras Sharma, Anthony Sicilia, Katherine Atwell, Diane Litman, Malihe Alikhani,
• Abstract summary: General-purpose automatic speech recognition (ASR) systems do not always perform well in goal-oriented dialogue. We extend correction to tasks that have no prior user data and exhibit linguistic flexibility such as lexical and syntactic variations.
• Score: 27.56203179880491
• License:
• Abstract: General-purpose automatic speech recognition (ASR) systems do not always perform well in goal-oriented dialogue. Existing ASR correction methods rely on prior user data or named entities. We extend correction to tasks that have no prior user data and exhibit linguistic flexibility such as lexical and syntactic variations. We propose a novel context augmentation with a large language model and a ranking strategy that incorporates contextual information from the dialogue states of a goal-oriented conversational AI and its tasks. Our method ranks (1) n-best ASR hypotheses by their lexical and semantic similarity with context and (2) context by phonetic correspondence with ASR hypotheses. Evaluated in home improvement and cooking domains with real-world users, our method improves recall and F1 of correction by 34% and 16%, respectively, while maintaining precision and false positive rate. Users rated .8-1 point (out of 5) higher when our correction method worked properly, with no decrease due to false positives.

Related papers

• Keyword-Aware ASR Error Augmentation for Robust Dialogue State Tracking [17.96115263146684]
  We introduce a simple yet effective data augmentation method to improve robustness of Dialogue State Tracking model. Our method generates sufficient error patterns on keywords, leading to improved accuracy in noised and low-accuracy ASR environments.
  arXiv  Detail & Related papers  (2024-09-10T07:06:40Z)
• Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learning [6.363223418619587]
  We introduce Context Noise Representation Learning (CNRL) to enhance robustness against noisy context, ultimately improving dialogue speech recognition accuracy. Based on the evaluation of speech dialogues, our method shows superior results compared to baselines.
  arXiv  Detail & Related papers  (2024-08-12T10:21:09Z)
• Towards interfacing large language models with ASR systems using confidence measures and prompting [54.39667883394458]
  This work investigates post-hoc correction of ASR transcripts with large language models (LLMs) To avoid introducing errors into likely accurate transcripts, we propose a range of confidence-based filtering methods. Our results indicate that this can improve the performance of less competitive ASR systems.
  arXiv  Detail & Related papers  (2024-07-31T08:00:41Z)
• Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition [52.624909026294105]
  We propose a non-autoregressive speech error correction method. A Confidence Module measures the uncertainty of each word of the N-best ASR hypotheses. The proposed system reduces the error rate by 21% compared with the ASR model.
  arXiv  Detail & Related papers  (2024-06-29T17:56:28Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding [28.441725610692714]
  This paper focuses on learning utterance representations that are robust to ASR errors using a contrastive objective. Experiments on three benchmark datasets demonstrate the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2022-05-02T07:21:21Z)
• Sequence-level self-learning with multiple hypotheses [53.04725240411895]
  We develop new self-learning techniques with an attention-based sequence-to-sequence (seq2seq) model for automatic speech recognition (ASR) In contrast to conventional unsupervised learning approaches, we adopt the emphmulti-task learning (MTL) framework. Our experiment results show that our method can reduce the WER on the British speech data from 14.55% to 10.36% compared to the baseline model trained with the US English data only.
  arXiv  Detail & Related papers  (2021-12-10T20:47:58Z)
• Improving Readability for Automatic Speech Recognition Transcription [50.86019112545596]
  We propose a novel NLP task called ASR post-processing for readability (APR) APR aims to transform the noisy ASR output into a readable text for humans and downstream tasks while maintaining the semantic meaning of the speaker. We compare fine-tuned models based on several open-sourced and adapted pre-trained models with the traditional pipeline method.
  arXiv  Detail & Related papers  (2020-04-09T09:26:42Z)
• ASR Error Correction and Domain Adaptation Using Machine Translation [32.27379508770736]
  We propose a technique to perform domain adaptation for ASR error correction via machine translation. We observe absolute improvement in word error rate and 4 point absolute improvement in BLEU score in Google ASR output.
  arXiv  Detail & Related papers  (2020-03-13T20:05:38Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.