Full-text Error Correction for Chinese Speech Recognition with Large Language Model

• URL: http://arxiv.org/abs/2409.07790v1
• Date: Thu, 12 Sep 2024 06:50:45 GMT
• Title: Full-text Error Correction for Chinese Speech Recognition with Large Language Model
• Authors: Zhiyuan Tang, Dong Wang, Shen Huang, Shidong Shang,
• Abstract summary: Large Language Models (LLMs) have demonstrated substantial potential for error correction in Automatic Speech Recognition (ASR) This paper investigates the effectiveness of LLMs for error correction in full-text generated by ASR systems from longer speech recordings.
• Score: 11.287933170894311
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Language Models (LLMs) have demonstrated substantial potential for error correction in Automatic Speech Recognition (ASR). However, most research focuses on utterances from short-duration speech recordings, which are the predominant form of speech data for supervised ASR training. This paper investigates the effectiveness of LLMs for error correction in full-text generated by ASR systems from longer speech recordings, such as transcripts from podcasts, news broadcasts, and meetings. First, we develop a Chinese dataset for full-text error correction, named ChFT, utilizing a pipeline that involves text-to-speech synthesis, ASR, and error-correction pair extractor. This dataset enables us to correct errors across contexts, including both full-text and segment, and to address a broader range of error types, such as punctuation restoration and inverse text normalization, thus making the correction process comprehensive. Second, we fine-tune a pre-trained LLM on the constructed dataset using a diverse set of prompts and target formats, and evaluate its performance on full-text error correction. Specifically, we design prompts based on full-text and segment, considering various output formats, such as directly corrected text and JSON-based error-correction pairs. Through various test settings, including homogeneous, up-to-date, and hard test sets, we find that the fine-tuned LLMs perform well in the full-text setting with different prompts, each presenting its own strengths and weaknesses. This establishes a promising baseline for further research. The dataset is available on the website.

Related papers

• VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning [64.56272011710735]
  We propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the large language models (LLMs) backbone. Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks.
  arXiv  Detail & Related papers  (2024-10-23T00:36:06Z)
• Spontaneous Informal Speech Dataset for Punctuation Restoration [0.8517406772939293]
  We introduce SponSpeech, a punctuation restoration dataset derived from informal speech sources. Our filtering pipeline examines the quality of both speech audio and transcription text. We also carefully construct a challenging" test set, aimed at evaluating models' ability to leverage audio information to predict otherwise grammatically ambiguous punctuation.
  arXiv  Detail & Related papers  (2024-09-17T14:43:14Z)
• Robustness of LLMs to Perturbations in Text [2.0670689746336]
  Large language models (LLMs) have shown impressive performance, but can they handle the inevitable noise in real-world data? This work tackles this critical question by investigating LLMs' resilience against morphological variations in text. Our findings show that contrary to popular beliefs, generative LLMs are quiet robust to noisy perturbations in text.
  arXiv  Detail & Related papers  (2024-07-12T04:50:17Z)
• Pinyin Regularization in Error Correction for Chinese Speech Recognition with Large Language Models [11.287933170894311]
  We construct a specialized benchmark dataset aimed at error correction for Chinese ASR with 724K hypotheses-transcription pairs. We propose a method of Pinyin regularization for prompts, which involves the transcription of Pinyin directly from text hypotheses.
  arXiv  Detail & Related papers  (2024-07-02T03:16:47Z)
• Detecting Errors through Ensembling Prompts (DEEP): An End-to-End LLM Framework for Detecting Factual Errors [11.07539342949602]
  We propose an end-to-end framework for detecting factual errors in text summarization. Our framework uses a diverse set of LLM prompts to identify factual inconsistencies. We calibrate the ensembled models to produce empirically accurate probabilities that a text is factually consistent or free of hallucination.
  arXiv  Detail & Related papers  (2024-06-18T18:59:37Z)
• Code-Switching Text Generation and Injection in Mandarin-English ASR [57.57570417273262]
  We investigate text generation and injection for improving the performance of an industry commonly-used streaming model, Transformer-Transducer (T-T) We first propose a strategy to generate code-switching text data and then investigate injecting generated text into T-T model explicitly by Text-To-Speech (TTS) conversion or implicitly by tying speech and text latent spaces. Experimental results on the T-T model trained with a dataset containing 1,800 hours of real Mandarin-English code-switched speech show that our approaches to inject generated code-switching text significantly boost the performance of T-T models.
  arXiv  Detail & Related papers  (2023-03-20T09:13:27Z)
• ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models [82.63962107729994]
  Any-Shot Data-to-Text (ASDOT) is a new approach flexibly applicable to diverse settings. It consists of two steps, data disambiguation and sentence fusion. Experimental results show that ASDOT consistently achieves significant improvement over baselines.
  arXiv  Detail & Related papers  (2022-10-09T19:17:43Z)
• Factual Error Correction for Abstractive Summaries Using Entity Retrieval [57.01193722520597]
  We propose an efficient factual error correction system RFEC based on entities retrieval post-editing process. RFEC retrieves the evidence sentences from the original document by comparing the sentences with the target summary. Next, RFEC detects the entity-level errors in the summaries by considering the evidence sentences and substitutes the wrong entities with the accurate entities from the evidence sentences.
  arXiv  Detail & Related papers  (2022-04-18T11:35:02Z)
• On the Robustness of Language Encoders against Grammatical Errors [66.05648604987479]
  We collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. Results confirm that the performance of all tested models is affected but the degree of impact varies.
  arXiv  Detail & Related papers  (2020-05-12T11:01:44Z)
• Towards Minimal Supervision BERT-based Grammar Error Correction [81.90356787324481]
  We try to incorporate contextual information from pre-trained language model to leverage annotation and benefit multilingual scenarios. Results show strong potential of Bidirectional Representations from Transformers (BERT) in grammatical error correction task.
  arXiv  Detail & Related papers  (2020-01-10T15:45:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.