Augmenting Automatic Speech Recognition Models with Disfluency Detection

• URL: http://arxiv.org/abs/2409.10177v2
• Date: Tue, 17 Sep 2024 06:30:03 GMT
• Title: Augmenting Automatic Speech Recognition Models with Disfluency Detection
• Authors: Robin Amann, Zhaolin Li, Barbara Bruno, Jan Niehues,
• Abstract summary: Speech disfluency commonly occurs in conversational and spontaneous speech. Current research mainly focuses on detecting disfluencies within transcripts, overlooking their exact location and duration in the speech. We present an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies.
• Score: 12.45703869323415
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Speech disfluency commonly occurs in conversational and spontaneous speech. However, standard Automatic Speech Recognition (ASR) models struggle to accurately recognize these disfluencies because they are typically trained on fluent transcripts. Current research mainly focuses on detecting disfluencies within transcripts, overlooking their exact location and duration in the speech. Additionally, previous work often requires model fine-tuning and addresses limited types of disfluencies. In this work, we present an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies. We first demonstrate that ASR models have difficulty transcribing speech disfluencies. Next, this work proposes a modified Connectionist Temporal Classification(CTC)-based forced alignment algorithm from \cite{kurzinger2020ctc} to predict word-level timestamps while effectively capturing disfluent speech. Additionally, we develop a model to classify alignment gaps between timestamps as either containing disfluent speech or silence. This model achieves an accuracy of 81.62% and an F1-score of 80.07%. We test the augmentation pipeline of alignment gap detection and classification on a disfluent dataset. Our results show that we captured 74.13% of the words that were initially missed by the transcription, demonstrating the potential of this pipeline for downstream tasks.

Related papers

• Time and Tokens: Benchmarking End-to-End Speech Dysfluency Detection [6.82179836198974]
  Speech dysfluency modeling is a task to detect dysfluencies in speech, such as repetition, block, insertion, replacement, and deletion. In this work, we revisit this problem from a new perspective: tokenizing dysfluencies and modeling the detection problem as a token-based automatic speech recognition problem. We propose rule-based speech and text dysfluency simulators and develop VCTK-token, and then develop a Whisper-like seq2seq architecture to build a new benchmark with decent performance.
  arXiv  Detail & Related papers  (2024-09-20T15:35:32Z)
• YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection [5.42845980208244]
  YOLO-Stutter is a first end-to-end method that detects dysfluencies in a time-accurate manner. VCTK-Stutter and VCTK-TTS simulate natural spoken dysfluencies including repetition, block, missing, replacement, and prolongation.
  arXiv  Detail & Related papers  (2024-08-27T11:31:12Z)
• Improving Sampling Methods for Fine-tuning SentenceBERT in Text Streams [49.3179290313959]
  This study explores the efficacy of seven text sampling methods designed to selectively fine-tune language models. We precisely assess the impact of these methods on fine-tuning the SBERT model using four different loss functions. Our findings indicate that Softmax loss and Batch All Triplets loss are particularly effective for text stream classification.
  arXiv  Detail & Related papers  (2024-03-18T23:41:52Z)
• Automatic Disfluency Detection from Untranscribed Speech [25.534535098405602]
  Stuttering is a speech disorder characterized by a high rate of disfluencies. automatic disfluency detection may help in treatment planning for individuals who stutter. We investigate language, acoustic, and multimodal methods for frame-level automatic disfluency detection and categorization.
  arXiv  Detail & Related papers  (2023-11-01T21:36:39Z)
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context [55.564789967211844]
  An important desideratum of RALMs, is that retrieved information helps model performance when it is relevant. Recent work has shown that retrieval augmentation can sometimes have a negative effect on performance.
  arXiv  Detail & Related papers  (2023-10-02T18:52:35Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Adversarial Training For Low-Resource Disfluency Correction [50.51901599433536]
  We propose an adversarially-trained sequence-tagging model for Disfluency Correction (DC) We show the benefit of our proposed technique, which crucially depends on synthetically generated disfluent data, by evaluating it for DC in three Indian languages. Our technique also performs well in removing stuttering disfluencies in ASR transcripts introduced by speech impairments.
  arXiv  Detail & Related papers  (2023-06-10T08:58:53Z)
• Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling [10.283092375534311]
  We propose a simple and effective modification of alignment graph construction using weighted Finite State Transducers. The proposed weakly-supervised approach alleviates the need for verbatim transcription of speech disfluencies for forced alignment. Our evaluation on a corrupted version of the TIMIT test set and the UCLASS dataset shows significant improvements.
  arXiv  Detail & Related papers  (2023-05-30T09:57:36Z)
• DisfluencyFixer: A tool to enhance Language Learning through Speech To Speech Disfluency Correction [50.51901599433536]
  DisfluencyFixer is a tool that performs speech-to-speech disfluency correction in English and Hindi. Our proposed system removes disfluencies from input speech and returns fluent speech as output.
  arXiv  Detail & Related papers  (2023-05-26T14:13:38Z)
• End-to-End Speech Recognition and Disfluency Removal [15.910282983166024]
  This paper investigates the task of end-to-end speech recognition and disfluency removal. We show that end-to-end models do learn to directly generate fluent transcripts. We propose two new metrics that can be used for evaluating integrated ASR and disfluency models.
  arXiv  Detail & Related papers  (2020-09-22T03:11:37Z)
• Wake Word Detection with Alignment-Free Lattice-Free MMI [66.12175350462263]
  Always-on spoken language interfaces, e.g. personal digital assistants, rely on a wake word to start processing spoken input. We present novel methods to train a hybrid DNN/HMM wake word detection system from partially labeled training data. We evaluate our methods on two real data sets, showing 50%--90% reduction in false rejection rates at pre-specified false alarm rates over the best previously published figures.
  arXiv  Detail & Related papers  (2020-05-17T19:22:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.