Crossmodal ASR Error Correction with Discrete Speech Units

• URL: http://arxiv.org/abs/2405.16677v2
• Date: Fri, 13 Sep 2024 01:56:05 GMT
• Title: Crossmodal ASR Error Correction with Discrete Speech Units
• Authors: Yuanchao Li, Pinzhen Chen, Peter Bell, Catherine Lai,
• Abstract summary: We propose a post-ASR processing approach for ASR Error Correction (AEC) We explore pre-training and fine-tuning strategies and uncover an ASR domain discrepancy phenomenon. We propose the incorporation of discrete speech units to align with and enhance the word embeddings for improving AEC quality.
• Score: 16.58209270191005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: ASR remains unsatisfactory in scenarios where the speaking style diverges from that used to train ASR systems, resulting in erroneous transcripts. To address this, ASR Error Correction (AEC), a post-ASR processing approach, is required. In this work, we tackle an understudied issue: the Low-Resource Out-of-Domain (LROOD) problem, by investigating crossmodal AEC on very limited downstream data with 1-best hypothesis transcription. We explore pre-training and fine-tuning strategies and uncover an ASR domain discrepancy phenomenon, shedding light on appropriate training schemes for LROOD data. Moreover, we propose the incorporation of discrete speech units to align with and enhance the word embeddings for improving AEC quality. Results from multiple corpora and several evaluation metrics demonstrate the feasibility and efficacy of our proposed AEC approach on LROOD data as well as its generalizability and superiority on large-scale data. Finally, a study on speech emotion recognition confirms that our model produces ASR error-robust transcripts suitable for downstream applications.

Related papers

• Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation [73.9145653659403]
  We show that Generative Error Correction models struggle to generalize beyond the specific types of errors encountered during training. We propose DARAG, a novel approach designed to improve GEC for ASR in in-domain (ID) and OOD scenarios. Our approach is simple, scalable, and both domain- and language-agnostic.
  arXiv  Detail & Related papers  (2024-10-17T04:00:29Z)
• Towards interfacing large language models with ASR systems using confidence measures and prompting [54.39667883394458]
  This work investigates post-hoc correction of ASR transcripts with large language models (LLMs) To avoid introducing errors into likely accurate transcripts, we propose a range of confidence-based filtering methods. Our results indicate that this can improve the performance of less competitive ASR systems.
  arXiv  Detail & Related papers  (2024-07-31T08:00:41Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• RED-ACE: Robust Error Detection for ASR using Confidence Embeddings [5.4693121539705984]
  We propose to utilize the ASR system's word-level confidence scores for improving AED performance. We add an ASR Confidence Embedding layer to the AED model's encoder, allowing us to jointly encode the confidence scores and the transcribed text into a contextualized representation.
  arXiv  Detail & Related papers  (2022-03-14T15:13:52Z)
• Attention-based Multi-hypothesis Fusion for Speech Summarization [83.04957603852571]
  Speech summarization can be achieved by combining automatic speech recognition (ASR) and text summarization (TS) ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary.
  arXiv  Detail & Related papers  (2021-11-16T03:00:29Z)
• Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition [71.96870151495536]
  We propose a novel adversarial reprogramming (AR) approach for low-resource spoken command recognition (SCR) The AR procedure aims to modify the acoustic signals (from the target domain) to repurpose a pretrained SCR model. We evaluate the proposed AR-SCR system on three low-resource SCR datasets, including Arabic, Lithuanian, and dysarthric Mandarin speech.
  arXiv  Detail & Related papers  (2021-10-08T05:07:35Z)
• An Approach to Improve Robustness of NLP Systems against ASR Errors [39.57253455717825]
  Speech-enabled systems typically first convert audio to text through an automatic speech recognition model and then feed the text to downstream natural language processing modules. The errors of the ASR system can seriously downgrade the performance of the NLP modules. Previous work has shown it is effective to employ data augmentation methods to solve this problem by injecting ASR noise during the training process.
  arXiv  Detail & Related papers  (2021-03-25T05:15:43Z)
• Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks [10.723935272906461]
  Adversarial training of end-to-end (E2E) ASR systems using generative adversarial networks (GAN) has recently been explored. We introduce a novel framework for fine-tuning a pre-trained ASR model using the GAN objective. Our proposed approach outperforms baselines and conventional GAN-based adversarial models.
  arXiv  Detail & Related papers  (2021-03-10T17:40:48Z)
• ASR Error Correction and Domain Adaptation Using Machine Translation [32.27379508770736]
  We propose a technique to perform domain adaptation for ASR error correction via machine translation. We observe absolute improvement in word error rate and 4 point absolute improvement in BLEU score in Google ASR output.
  arXiv  Detail & Related papers  (2020-03-13T20:05:38Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.