Replacing Human Audio with Synthetic Audio for On-device Unspoken Punctuation Prediction

• URL: http://arxiv.org/abs/2010.10203v2
• Date: Wed, 10 Feb 2021 21:01:35 GMT
• Title: Replacing Human Audio with Synthetic Audio for On-device Unspoken Punctuation Prediction
• Authors: Daria Soboleva, Ondrej Skopek, M\'arius \v{S}ajgal\'ik, Victor C\u{a}rbune, Felix Weissenberger, Julia Proskurnia, Bogdan Prisacari, Daniel Valcarce, Justin Lu, Rohit Prabhavalkar, Balint Miklos
• Abstract summary: We present a novel multi-modal unspoken punctuation prediction system for the English language which combines acoustic and text features. We demonstrate for the first time, that by relying exclusively on synthetic data generated using a prosody-aware text-to-speech system, we can outperform a model trained with expensive human audio recordings on the unspoken punctuation prediction problem.
• Score: 10.516452073178511
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel multi-modal unspoken punctuation prediction system for the English language which combines acoustic and text features. We demonstrate for the first time, that by relying exclusively on synthetic data generated using a prosody-aware text-to-speech system, we can outperform a model trained with expensive human audio recordings on the unspoken punctuation prediction problem. Our model architecture is well suited for on-device use. This is achieved by leveraging hash-based embeddings of automatic speech recognition text output in conjunction with acoustic features as input to a quasi-recurrent neural network, keeping the model size small and latency low.

Related papers

• SONAR: A Synthetic AI-Audio Detection Framework and Benchmark [59.09338266364506]
  SONAR is a synthetic AI-Audio Detection Framework and Benchmark. It aims to provide a comprehensive evaluation for distinguishing cutting-edge AI-synthesized auditory content. It is the first framework to uniformly benchmark AI-audio detection across both traditional and foundation model-based deepfake detection systems.
  arXiv  Detail & Related papers  (2024-10-06T01:03:42Z)
• Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems [55.99999020778169]
  We study a function that can predict the forthcoming words and estimate the time remaining until the end of an utterance. We develop a cross-attention-based algorithm that incorporates both acoustic and linguistic information. Results demonstrate the proposed model's ability to predict upcoming words and estimate future EOU events up to 300ms prior to the actual EOU.
  arXiv  Detail & Related papers  (2024-09-30T06:29:58Z)
• Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models [64.14812728562596]
  We present a method for reprogramming pre-trained audio-driven talking face synthesis models to operate in a text-driven manner. We can easily generate face videos that articulate the provided textual sentences.
  arXiv  Detail & Related papers  (2023-06-28T08:22:53Z)
• CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models [50.42886595228255]
  We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining model.
  arXiv  Detail & Related papers  (2023-06-16T05:42:01Z)
• Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model [13.572330725278066]
  A novel point of the proposed method is the direct use of the SSL model to obtain embedding vectors from speech representations trained with a large amount of data. The disentangled embeddings will enable us to achieve better reproduction performance for unseen speakers and rhythm transfer conditioned by different speeches.
  arXiv  Detail & Related papers  (2023-04-24T10:15:58Z)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers [90.83782600932567]
  We develop NaturalSpeech 2, a TTS system that leverages a neural audio predictor with residual vectorizers to get the quantized latent vectors. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, synthesis, and voice quality in a zero-shot setting.
  arXiv  Detail & Related papers  (2023-04-18T16:31:59Z)
• End-to-end Speech-to-Punctuated-Text Recognition [23.44236710364419]
  punctuation marks are important for the readability of the speech recognition results. Conventional automatic speech recognition systems do not produce punctuation marks. We propose an end-to-end model that takes speech as input and outputs punctuated texts.
  arXiv  Detail & Related papers  (2022-07-07T08:58:01Z)
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages [58.43299730989809]
  We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task. This process stands on its own, or can be applied as low-cost second-stage pre-training.
  arXiv  Detail & Related papers  (2022-05-02T17:59:02Z)
• Revisiting joint decoding based multi-talker speech recognition with DNN acoustic model [34.061441900912136]
  We argue that such a scheme is sub-optimal and propose a principled solution that decodes all speakers jointly. We modify the acoustic model to predict joint state posteriors for all speakers, enabling the network to express uncertainty about the attribution of parts of the speech signal to the speakers.
  arXiv  Detail & Related papers  (2021-10-31T09:28:04Z)
• End to End ASR System with Automatic Punctuation Insertion [0.0]
  We propose a method to generate punctuated transcript for the TEDLIUM dataset using transcripts available from ted.com. We also propose an end-to-end ASR system that outputs words and punctuations concurrently from speech signals.
  arXiv  Detail & Related papers  (2020-12-03T15:46:43Z)
• Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement [31.33429812278942]
  The proposed end-to-end speech synthesis model uses both speaker embedding and noise representation as conditional inputs to model speaker and noise information respectively. Experimental results show that the speech generated by the proposed approach has better subjective evaluation results than the method directly fine-tuning multi-speaker speech synthesis model.
  arXiv  Detail & Related papers  (2020-05-26T06:14:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.