UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling

• URL: http://arxiv.org/abs/2606.31128v1
• Date: Tue, 30 Jun 2026 04:46:45 GMT
• Title: UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling
• Abstract summary: Speech editing aims to modify specific portions of an utterance while preserving the remaining speech.<n>We propose UniSAE, a unified speech attribute editing framework which supports composable speaker, emotion and content editing from sub-phoneme to word level within a single architecture.<n> Experimental results demonstrate that the proposed unified framework supports precise speaker and emotion control, content editing at multiple granularities, and joint modification of all three attributes within a single framework.
• Score: 31.829301124201226
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Speech editing aims to modify specific portions of an utterance while preserving the remaining speech. Existing approaches primarily focus on word-level content modification and typically treat content, speaker, and emotion editing as separate tasks, limiting both editing granularity and flexibility. We propose UniSAE, a unified speech attribute editing framework which supports composable speaker, emotion and content editing from sub-phoneme to word level within a single architecture. UniSAE introduces a Discrete Phonetic PosteriorGram (DPPG) representation that factorizes speech content into discrete tokens encoding phoneme identity, pronunciation variants, and duration, enabling direct phoneme- and sub-phoneme-level editing. For higher-level modifications, an autoregressive content transformer predicts edited DPPG sequences for word-level content editing. The edited sequences are rendered into speech by a diffusion-based acoustic decoder, conditioned on disentangled speaker and emotion representations. Experimental results demonstrate that the proposed unified framework supports precise speaker and emotion control, content editing at multiple granularities, and joint modification of all three attributes within a single framework.

Related papers

• Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing [10.49533454106105]
  Speech inpainting restores missing segments, whereas speech editing replaces spoken content according to an edited transcript.<n>We introduce SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing over hierarchical tokens.<n>On the RealEdit benchmark, SIEDD achieves the best overall speech-editing performance among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-05T18:57:34Z)
• dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model [25.837532849844436]
  We study a precise and explicit interface for speech editing with XML-style tags.<n>We instantiate the interface in dots$.$tts$.$edit, an editor adapted from the continuous autoregressive dots$.$tts foundation model.<n>We also introduce doteBench, a bilingual evaluation suite that measures precise instruction following, local preservation, and audio quality.
  arXiv  Detail & Related papers  (2026-08-02T15:03:37Z)
• Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation [56.92841782969847]
  We introduce a novel task called language-guided joint audio-visual editing. Given an audio and image pair of a sounding event, this task aims at generating new audio-visual content by editing the given sounding event conditioned on the language guidance. We propose a new diffusion-based framework for joint audio-visual editing and introduce two key ideas.
  arXiv  Detail & Related papers  (2024-10-09T22:02:30Z)
• FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistency [40.95700389032375]
  Text-based speech editing (TSE) allows users to edit speech by modifying the corresponding text directly without altering the original recording.<n>Current TSE techniques often focus on minimizing discrepancies between generated speech and reference within edited regions during training to achieve fluent TSE performance.<n>We propose a new fluency speech editing scheme based on our previous textitFluentEditor model, termed textittextbfFluentEditor2.
  arXiv  Detail & Related papers  (2024-09-28T10:18:35Z)
• DiffEditor: Enhancing Speech Editing with Semantic Enrichment and Acoustic Consistency [20.3466261946094]
  We introduce DiffEditor, a novel speech editing model designed to enhance performance in OOD text scenarios. We enrich the semantic information of phoneme embeddings by integrating word embeddings extracted from a pretrained language model. We propose a first-order loss function to promote smoother transitions at editing boundaries and enhance the overall fluency of the edited speech.
  arXiv  Detail & Related papers  (2024-09-19T07:11:54Z)
• Voice Attribute Editing with Text Prompt [48.48628304530097]
  This paper introduces a novel task: voice attribute editing with text prompt.<n>The goal is to make relative modifications to voice attributes according to the actions described in the text prompt.<n>To solve this task, VoxEditor, an end-to-end generative model, is proposed.
  arXiv  Detail & Related papers  (2024-04-13T00:07:40Z)
• VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing [20.446399170808682]
  We present VoiceShop, a novel speech-to-speech framework. It can modify multiple attributes of speech, such as age, gender, accent, and speech style, in a single forward pass. The magnitude of the conversion effect is weak, there is no zero-shot capability for out-of-distribution speakers, or the synthesized outputs exhibit undesirable timbre leakage.
  arXiv  Detail & Related papers  (2024-04-10T01:33:08Z)
• FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency [44.7425844190807]
  Text-based speech editing (TSE) techniques are designed to enable users to edit the output audio by modifying the input text transcript instead of the audio itself. We propose a fluency speech editing model, termed textitFluentEditor, by considering fluency-aware training criterion in the TSE training. The subjective and objective experimental results on VCTK demonstrate that our textitFluentEditor outperforms all advanced baselines in terms of naturalness and fluency.
  arXiv  Detail & Related papers  (2023-09-21T01:58:01Z)
• Emotion Selectable End-to-End Text-based Speech Editing [63.346825713704625]
  Emo-CampNet (emotion CampNet) is an emotion-selectable text-based speech editing model. It can effectively control the emotion of the generated speech in the process of text-based speech editing. It can also edit unseen speakers' speech.
  arXiv  Detail & Related papers  (2022-12-20T12:02:40Z)
• SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data [100.46303484627045]
  We propose a cross-modal Speech and Language Model (SpeechLM) to align speech and text pre-training with a pre-defined unified representation. Specifically, we introduce two alternative discrete tokenizers to bridge the speech and text modalities. We evaluate SpeechLM on various spoken language processing tasks including speech recognition, speech translation, and universal representation evaluation framework SUPERB.
  arXiv  Detail & Related papers  (2022-09-30T09:12:10Z)
• Textless Speech Emotion Conversion using Decomposed and Discrete Representations [49.55101900501656]
  We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion. First, we modify the speech content by translating the content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder.
  arXiv  Detail & Related papers  (2021-11-14T18:16:42Z)
• Bridging the Modality Gap for Speech-to-Text Translation [57.47099674461832]
  End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously. We propose a Speech-to-Text Adaptation for Speech Translation model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text.
  arXiv  Detail & Related papers  (2020-10-28T12:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.