Automatic Restoration of Diacritics for Speech Data Sets

• URL: http://arxiv.org/abs/2311.10771v2
• Date: Sun, 7 Apr 2024 00:48:10 GMT
• Title: Automatic Restoration of Diacritics for Speech Data Sets
• Authors: Sara Shatnawi, Sawsan Alqahtani, Hanan Aldarmaki,
• Abstract summary: We explore the possibility of improving the performance of automatic diacritic restoration when applied to speech data by utilizing parallel spoken utterances. We use the pre-trained Whisper ASR model fine-tuned on relatively small amounts of diacritized Arabic speech data to produce rough diacritized transcripts for the speech utterances. The proposed framework consistently improves diacritic restoration performance compared to text-only baselines.
• Score: 1.81336359426598
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic text-based diacritic restoration models generally have high diacritic error rates when applied to speech transcripts as a result of domain and style shifts in spoken language. In this work, we explore the possibility of improving the performance of automatic diacritic restoration when applied to speech data by utilizing parallel spoken utterances. In particular, we use the pre-trained Whisper ASR model fine-tuned on relatively small amounts of diacritized Arabic speech data to produce rough diacritized transcripts for the speech utterances, which we then use as an additional input for diacritic restoration models. The proposed framework consistently improves diacritic restoration performance compared to text-only baselines. Our results highlight the inadequacy of current text-based diacritic restoration models for speech data sets and provide a new baseline for speech-based diacritic restoration.

Related papers

• Spontaneous Informal Speech Dataset for Punctuation Restoration [0.8517406772939293]
  We introduce SponSpeech, a punctuation restoration dataset derived from informal speech sources. Our filtering pipeline examines the quality of both speech audio and transcription text. We also carefully construct a challenging" test set, aimed at evaluating models' ability to leverage audio information to predict otherwise grammatically ambiguous punctuation.
  arXiv  Detail & Related papers  (2024-09-17T14:43:14Z)
• LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models [58.790604613878216]
  We introduce a LibriSpeech-PC benchmark designed to assess the punctuation and capitalization prediction capabilities of end-to-end ASR models. The benchmark includes a LibriSpeech-PC dataset with restored punctuation and capitalization, a novel evaluation metric called Punctuation Error Rate (PER) that focuses on punctuation marks, and initial baseline models.
  arXiv  Detail & Related papers  (2023-10-04T16:23:37Z)
• EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis [49.04496602282718]
  We introduce Expresso, a high-quality expressive speech dataset for textless speech synthesis. This dataset includes both read speech and improvised dialogues rendered in 26 spontaneous expressive styles. We evaluate resynthesis quality with automatic metrics for different self-supervised discrete encoders.
  arXiv  Detail & Related papers  (2023-08-10T17:41:19Z)
• Boosting Punctuation Restoration with Data Generation and Reinforcement Learning [70.26450819702728]
  Punctuation restoration is an important task in automatic speech recognition (ASR) The discrepancy between written punctuated texts and ASR texts limits the usability of written texts in training punctuation restoration systems for ASR texts. This paper proposes a reinforcement learning method to exploit in-topic written texts and recent advances in large pre-trained generative language models to bridge this gap.
  arXiv  Detail & Related papers  (2023-07-24T17:22:04Z)
• High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Units [69.06657692891447]
  We propose a novel AVO method leveraging the learning objective of self-supervised discrete speech unit prediction. Experimental results show that our proposed method achieves remarkable lip-speech synchronization and high speech quality.
  arXiv  Detail & Related papers  (2023-06-29T15:02:22Z)
• Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling [10.283092375534311]
  We propose a simple and effective modification of alignment graph construction using weighted Finite State Transducers. The proposed weakly-supervised approach alleviates the need for verbatim transcription of speech disfluencies for forced alignment. Our evaluation on a corrupted version of the TIMIT test set and the UCLASS dataset shows significant improvements.
  arXiv  Detail & Related papers  (2023-05-30T09:57:36Z)
• Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations [51.89856133895233]
  Speech restoration (SR) is a task of converting degraded speech signals into high-quality ones. In this study, we propose a robust SR model called Miipher, and apply Miipher to a new SR application. To make our SR model robust against various degradation, we use (i) a speech representation extracted from w2v-BERT for the input feature, and (ii) a text representation extracted from transcripts via PnG-BERT as a linguistic conditioning feature.
  arXiv  Detail & Related papers  (2023-03-03T01:57:16Z)
• Diacritic Recognition Performance in Arabic ASR [2.28438857884398]
  We present an analysis of diacritic recognition performance in Arabic Automatic Speech Recognition systems. Current state-of-the-art ASR models do not produce full diacritization in their output.
  arXiv  Detail & Related papers  (2023-02-27T18:27:42Z)
• Towards zero-shot Text-based voice editing using acoustic context conditioning, utterance embeddings, and reference encoders [14.723225542605105]
  Text-based voice editing (TBVE) uses synthetic output from text-to-speech (TTS) systems to replace words in an original recording. Recent work has used neural models to produce edited speech similar to the original speech in terms of clarity, speaker identity, and prosody. This work focuses on the zero-shot approach which avoids finetuning altogether.
  arXiv  Detail & Related papers  (2022-10-28T10:31:44Z)
• Label-Synchronous Speech-to-Text Alignment for ASR Using Forward and Backward Transformers [49.403414751667135]
  This paper proposes a novel label-synchronous speech-to-text alignment technique for automatic speech recognition (ASR) The proposed method re-defines the speech-to-text alignment as a label-synchronous text mapping problem. Experiments using the corpus of spontaneous Japanese (CSJ) demonstrate that the proposed method provides an accurate utterance-wise alignment.
  arXiv  Detail & Related papers  (2021-04-21T03:05:12Z)
• A Multitask Learning Approach for Diacritic Restoration [21.288912928687186]
  In many languages like Arabic, diacritics are used to specify pronunciations as well as meanings. Such diacritics are often omitted in written text, increasing the number of possible pronunciations and meanings for a word. We use Arabic as a case study since it has sufficient data resources for tasks that we consider in our joint modeling.
  arXiv  Detail & Related papers  (2020-06-07T01:20:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.