Multi-stage Large Language Model Correction for Speech Recognition

• URL: http://arxiv.org/abs/2310.11532v2
• Date: Mon, 17 Jun 2024 15:21:50 GMT
• Title: Multi-stage Large Language Model Correction for Speech Recognition
• Authors: Jie Pu, Thai-Son Nguyen, Sebastian Stüker,
• Abstract summary: We propose a novel multi-stage approach that utilizes uncertainty estimation of ASR outputs and reasoning capability of large language models (LLMs) Our experimental results demonstrate the effectiveness of the proposed method by showing 10% 20% relative improvement in WER over competitive ASR systems.
• Score: 10.995600950995021
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this paper, we investigate the usage of large language models (LLMs) to improve the performance of competitive speech recognition systems. Different from previous LLM-based ASR error correction methods, we propose a novel multi-stage approach that utilizes uncertainty estimation of ASR outputs and reasoning capability of LLMs. Specifically, the proposed approach has two stages: the first stage is about ASR uncertainty estimation and exploits N-best list hypotheses to identify less reliable transcriptions; The second stage works on these identified transcriptions and performs LLM-based corrections. This correction task is formulated as a multi-step rule-based LLM reasoning process, which uses explicitly written rules in prompts to decompose the task into concrete reasoning steps. Our experimental results demonstrate the effectiveness of the proposed method by showing 10% ~ 20% relative improvement in WER over competitive ASR systems -- across multiple test domains and in zero-shot settings.

Related papers

• Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification [52.095460362197336]
  Large language models (LLMs) struggle with consistent and accurate reasoning. LLMs are trained primarily on correct solutions, reducing their ability to detect and learn from errors. We propose a novel collaborative method integrating Chain-of-Thought (CoT) and Program-of-Thought (PoT) solutions for verification.
  arXiv  Detail & Related papers  (2024-10-05T05:21:48Z)
• Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model [50.339632513018934]
  supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences. We critically examine this hypothesis within the scope of cross-lingual generation tasks. We introduce a novel training-free alignment method named PreTTY, which employs minimal task-related prior tokens.
  arXiv  Detail & Related papers  (2024-04-25T17:19:36Z)
• Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment [32.12998469814097]
  A novel causal prompting method based on front-door adjustment is proposed to effectively mitigate Large Language Models (LLMs) biases. Experimental results show that the proposed causal prompting approach achieves excellent performance across seven natural language processing datasets.
  arXiv  Detail & Related papers  (2024-03-05T07:47:34Z)
• Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks [68.79880423713597]
  We introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts.
  arXiv  Detail & Related papers  (2024-01-05T17:58:10Z)
• Correction Focused Language Model Training for Speech Recognition [14.246583065323192]
  We introduce a novel correction focused LM training approach which aims to prioritize ASR fallible words. The word-level ASR fallibility score is defined and shaped as a prior word distribution to guide the LM training. Compared with conventional LMs, correction focused training achieves up to relatively 5.5% word error rate (WER) reduction in sufficient text scenarios.
  arXiv  Detail & Related papers  (2023-10-17T05:10:39Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting [32.70214938434769]
  We explore the ability of large language models (LLMs) to act as speech recognition post-processors. We evaluate different prompting schemes, both zero- and few-shot in-context learning, and a novel task activation prompting method. We show that rescoring only by in-context learning with frozen LLMs achieves results that are competitive with rescoring by domain-tuned LMs.
  arXiv  Detail & Related papers  (2023-09-27T13:36:03Z)
• Leveraging Large Language Models for Exploiting ASR Uncertainty [16.740712975166407]
  Large language models must either rely on off-the-shelf automatic speech recognition systems for transcription, or be equipped with an in-built speech modality. We tackle speech-intent classification task, where a high word-error-rate can limit the LLM's ability to understand the spoken intent. We propose prompting the LLM with an n-best list of ASR hypotheses instead of only the error-prone 1-best hypothesis.
  arXiv  Detail & Related papers  (2023-09-09T17:02:33Z)
• On Language Model Integration for RNN Transducer based Speech Recognition [49.84285563767935]
  We study various ILM correction-based LM integration methods formulated in a common RNN-T framework. We provide a decoding interpretation on two major reasons for performance improvement with ILM correction. We also propose an exact-ILM training framework by extending the proof given in the hybrid autoregressive transducer.
  arXiv  Detail & Related papers  (2021-10-13T16:30:46Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.