dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model

• URL: http://arxiv.org/abs/2608.02673v2
• Date: Tue, 11 Aug 2026 17:29:29 GMT
• Title: dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model
• Abstract summary: We study a precise and explicit interface for speech editing with XML-style tags.<n>We instantiate the interface in dots$.$tts$.$edit, an editor adapted from the continuous autoregressive dots$.$tts foundation model.<n>We also introduce doteBench, a bilingual evaluation suite that measures precise instruction following, local preservation, and audio quality.
• Score: 25.837532849844436
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech editing for content creation requires precise control over both what an edit should do and where it should apply. Free-form natural language provides a flexible interface for expressing edit requests, but its ambiguity may leave the intended operation, parameters, or target region underspecified. We study a precise and explicit interface for speech editing: a transcript-grounded structural edit instruction with XML-style tags explicitly specifies typed operations and localizes them to transcript spans or boundaries. This semantic timeline avoids explicit timestamp alignment and provides an externally inspectable contract for compositional edits. We instantiate the interface in dots$.$tts$.$edit, an editor adapted from the continuous autoregressive dots$.$tts foundation model. Four representative speech-creation controls cover lexical content, affective expression, pitch and speaking-rate delivery, and temporal phrasing through text, emotion, prosody, and pause editing. Task-specific data pipelines construct operation- and scope-controlled pairs while retaining source-derived context outside each target region. We further introduce doteBench, a bilingual evaluation suite that measures precise instruction following, local preservation, and audio quality across the four controls and their composition. Experiments show leading overall instruction following and local preservation across its five editing categories, while audio quality remains comparable to existing open-source systems. Across three Seed-TTS-Eval shards, the model shows negligible differences from the base model in zero-shot TTS recognition error rate and speaker similarity.

Related papers

• TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models [49.44322041671119]
  emphinference-time speech scheduling is a novel task that places coupled speech and visual articulation within user-specified begin--end intervals.<n>We propose textbfTimeSteer, a training-free framework that localizes each utterance's source span.<n>TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.
  arXiv  Detail & Related papers  (2026-09-01T14:12:57Z)
• Vorch-Omni: Multi-Task Orchestration of Sight and Sound [51.39284238118125]
  Vorch- Omni is a unified framework for audio-visual synthesis based on an arbitrary-condition-to-arbitrary-language formulation.<n>It flexibly treats video and audio signals as either conditioning inputs or generation targets.<n>It supports over 10 tasks, including text-to-video, text-to-audio-video, image- and reference-conditioned generation, temporal extension, audio-driven generation, video transformation, and audio-visual editing.
  arXiv  Detail & Related papers  (2026-08-06T09:38:30Z)
• Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing [10.49533454106105]
  Speech inpainting restores missing segments, whereas speech editing replaces spoken content according to an edited transcript.<n>We introduce SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing over hierarchical tokens.<n>On the RealEdit benchmark, SIEDD achieves the best overall speech-editing performance among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-05T18:57:34Z)
• KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing [24.315720861063397]
  KGEdit is a structured semantic control framework for text-to-video (T2V) diffusion models.<n>We first construct an ambiguity-aware knowledge graph (AAKG) to disentangle and disambiguate the input prompt.<n>We then design a structured semantic injection module (SSIM) to inject semantic signals into key layers of the diffusion Transformer.
  arXiv  Detail & Related papers  (2026-05-28T07:31:22Z)
• Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing [55.211537893248675]
  We propose RVEDiT, an implicit Reasoning Video Editing DiT framework built around two complementary components.<n> RVEDiT consistently outperforms state-of-the-art baselines, with particularly strong gains on localized and compositional edits.
  arXiv  Detail & Related papers  (2026-05-23T17:22:14Z)
• InstructAudio: Unified speech and music generation with natural language instruction [52.76518112649456]
  InstructAudio is a unified framework that enables instruction-based control of acoustic attributes.<n>It supports expressive speech, music, and dialogue generation in English and Chinese.
  arXiv  Detail & Related papers  (2025-11-23T15:15:21Z)
• FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistency [40.95700389032375]
  Text-based speech editing (TSE) allows users to edit speech by modifying the corresponding text directly without altering the original recording.<n>Current TSE techniques often focus on minimizing discrepancies between generated speech and reference within edited regions during training to achieve fluent TSE performance.<n>We propose a new fluency speech editing scheme based on our previous textitFluentEditor model, termed textittextbfFluentEditor2.
  arXiv  Detail & Related papers  (2024-09-28T10:18:35Z)
• FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency [44.7425844190807]
  Text-based speech editing (TSE) techniques are designed to enable users to edit the output audio by modifying the input text transcript instead of the audio itself. We propose a fluency speech editing model, termed textitFluentEditor, by considering fluency-aware training criterion in the TSE training. The subjective and objective experimental results on VCTK demonstrate that our textitFluentEditor outperforms all advanced baselines in terms of naturalness and fluency.
  arXiv  Detail & Related papers  (2023-09-21T01:58:01Z)
• CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editing [67.96138567288197]
  This paper proposes a novel end-to-end text-based speech editing method called context-aware mask prediction network (CampNet) The model can simulate the text-based speech editing process by randomly masking part of speech and then predicting the masked region by sensing the speech context. It can solve unnatural prosody in the edited region and synthesize the speech corresponding to the unseen words in the transcript.
  arXiv  Detail & Related papers  (2022-02-21T02:05:14Z)
• Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration [62.75234183218897]
  We propose a one-stage context-aware framework to generate natural and coherent target speech without any training data of the speaker. We generate the mel-spectrogram of the edited speech with a transformer-based decoder. It outperforms a recent zero-shot TTS engine by a large margin.
  arXiv  Detail & Related papers  (2021-09-12T04:17:53Z)
• Context-Aware Prosody Correction for Text-Based Speech Editing [28.459695630420832]
  A major drawback of current systems is that edited recordings often sound unnatural because of prosody mismatches around edited regions. We propose a new context-aware method for more natural sounding text-based editing of speech.
  arXiv  Detail & Related papers  (2021-02-16T18:16:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.