FluentEditor: Text-based Speech Editing by Considering Acoustic and
Prosody Consistency
- URL: http://arxiv.org/abs/2309.11725v2
- Date: Fri, 22 Sep 2023 02:05:36 GMT
- Title: FluentEditor: Text-based Speech Editing by Considering Acoustic and
Prosody Consistency
- Authors: Rui Liu, Jiatian Xi, Ziyue Jiang and Haizhou Li
- Abstract summary: Text-based speech editing (TSE) techniques are designed to enable users to edit the output audio by modifying the input text transcript instead of the audio itself.
We propose a fluency speech editing model, termed textitFluentEditor, by considering fluency-aware training criterion in the TSE training.
The subjective and objective experimental results on VCTK demonstrate that our textitFluentEditor outperforms all advanced baselines in terms of naturalness and fluency.
- Score: 44.7425844190807
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Text-based speech editing (TSE) techniques are designed to enable users to
edit the output audio by modifying the input text transcript instead of the
audio itself. Despite much progress in neural network-based TSE techniques, the
current techniques have focused on reducing the difference between the
generated speech segment and the reference target in the editing region,
ignoring its local and global fluency in the context and original utterance. To
maintain the speech fluency, we propose a fluency speech editing model, termed
\textit{FluentEditor}, by considering fluency-aware training criterion in the
TSE training. Specifically, the \textit{acoustic consistency constraint} aims
to smooth the transition between the edited region and its neighboring acoustic
segments consistent with the ground truth, while the \textit{prosody
consistency constraint} seeks to ensure that the prosody attributes within the
edited regions remain consistent with the overall style of the original
utterance. The subjective and objective experimental results on VCTK
demonstrate that our \textit{FluentEditor} outperforms all advanced baselines
in terms of naturalness and fluency. The audio samples and code are available
at \url{https://github.com/Ai-S2-Lab/FluentEditor}.
Related papers
- Voice Attribute Editing with Text Prompt [48.48628304530097]
This paper introduces a novel task: voice attribute editing with text prompt.
The goal is to make relative modifications to voice attributes according to the actions described in the text prompt.
To solve this task, VoxEditor, an end-to-end generative model, is proposed.
arXiv Detail & Related papers (2024-04-13T00:07:40Z) - Text-only Domain Adaptation using Unified Speech-Text Representation in
Transducer [12.417314740402587]
We present a method to learn Unified Speech-Text Representation in Conformer Transducer(USTR-CT) to enable fast domain adaptation using the text-only corpus.
Experiments on adapting LibriSpeech to SPGISpeech show the proposed method reduces the word error rate(WER) by relatively 44% on the target domain.
arXiv Detail & Related papers (2023-06-07T00:33:02Z) - M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation [66.92823764664206]
We propose M-Adapter, a novel Transformer-based module, to adapt speech representations to text.
While shrinking the speech sequence, M-Adapter produces features desired for speech-to-text translation.
Our experimental results show that our model outperforms a strong baseline by up to 1 BLEU.
arXiv Detail & Related papers (2022-07-03T04:26:53Z) - CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech
Editing [67.96138567288197]
This paper proposes a novel end-to-end text-based speech editing method called context-aware mask prediction network (CampNet)
The model can simulate the text-based speech editing process by randomly masking part of speech and then predicting the masked region by sensing the speech context.
It can solve unnatural prosody in the edited region and synthesize the speech corresponding to the unseen words in the transcript.
arXiv Detail & Related papers (2022-02-21T02:05:14Z) - EdiTTS: Score-based Editing for Controllable Text-to-Speech [9.34612743192798]
EdiTTS is an off-the-shelf speech editing methodology based on score-based generative modeling for text-to-speech synthesis.
We apply coarse yet deliberate perturbations in the Gaussian prior space to induce desired behavior from the diffusion model.
Listening tests demonstrate that EdiTTS is capable of reliably generating natural-sounding audio that satisfies user-imposed requirements.
arXiv Detail & Related papers (2021-10-06T08:51:10Z) - Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration [62.75234183218897]
We propose a one-stage context-aware framework to generate natural and coherent target speech without any training data of the speaker.
We generate the mel-spectrogram of the edited speech with a transformer-based decoder.
It outperforms a recent zero-shot TTS engine by a large margin.
arXiv Detail & Related papers (2021-09-12T04:17:53Z) - Context-Aware Prosody Correction for Text-Based Speech Editing [28.459695630420832]
A major drawback of current systems is that edited recordings often sound unnatural because of prosody mismatches around edited regions.
We propose a new context-aware method for more natural sounding text-based editing of speech.
arXiv Detail & Related papers (2021-02-16T18:16:30Z) - Bridging the Modality Gap for Speech-to-Text Translation [57.47099674461832]
End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way.
Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously.
We propose a Speech-to-Text Adaptation for Speech Translation model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text.
arXiv Detail & Related papers (2020-10-28T12:33:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.