ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions

• URL: http://arxiv.org/abs/2607.05276v1
• Date: Mon, 06 Jul 2026 16:21:34 GMT
• Title: ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions
• Abstract summary: ProPS is a framework for generating distributions of speaker embeddings conditioned on natural language prompts.<n>We show that ProPS produces profile-conditioned distributions and generates x-vectors that preserve requested speaker attributes.
• Score: 17.76770343642986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speaker embeddings, or x-vectors, are widely used to represent speaker identity and speaker-related attributes, but existing embedding extractors are typically descriptive rather than generative: they map an observed speech segment to an x-vector, which is then used for downstream applications. We introduce ProPS, Prompted Profile Synthesis, a framework for generating distributions of speaker embeddings conditioned on natural language prompts such as "a thirties male speaker with an Indian accent". ProPS converts human-written profile descriptions into sentence embeddings and uses a mixture density network trained on a large-scale dataset to predict a Gaussian mixture model in the x-vector space. The model is trained by maximizing the likelihood that real speaker embeddings match the requested profile, and its generated distributions are evaluated by negative log-likelihood on held-out x-vectors and by attribute classification accuracies on sampled synthetic x-vectors. Experiments show that ProPS produces profile-conditioned distributions and generates x-vectors that preserve requested speaker attributes such as age, gender, accent, and prosodic characteristics. This design enables controllable speaker-profile synthesis for speech generation systems like Text-To-Speech (TTS) or Voice Conversion (VC) while anchoring generated distributions in observed speaker-embedding structure.

Related papers

• CoLMbo: Speaker Language Model for Descriptive Profiling [56.57669166980832]
  Speaker recognition systems are often limited to classification tasks and struggle to generate detailed speaker characteristics.<n>This paper introduces CoLMbo, a Speaker Language Model (SLM) that addresses these limitations by integrating a speaker encoder with prompt-based conditioning.<n>CoLMbo utilizes user-defined prompts to adapt dynamically to new speaker characteristics and provides customized descriptions.
  arXiv  Detail & Related papers  (2025-06-11T03:50:16Z)
• Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity [51.250471760075165]
  We propose a novel speaker embedding network that employs multiple sub-centers per speaker class during training.<n>This sub-center modeling allows the embedding to capture a broader range of speaker-specific variations while maintaining speaker classification performance.
  arXiv  Detail & Related papers  (2024-07-05T06:54:24Z)
• VoxGenesis: Unsupervised Discovery of Latent Speaker Manifold for Speech Synthesis [43.369048727268506]
  VoxGenesis is a novel unsupervised speech synthesis framework. It can discover a latent speaker manifold and meaningful voice editing directions without supervision. We show that VoxGenesis produces significantly more diverse and realistic speakers with distinct characteristics than the previous approaches.
  arXiv  Detail & Related papers  (2024-03-01T13:39:56Z)
• Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model [13.572330725278066]
  A novel point of the proposed method is the direct use of the SSL model to obtain embedding vectors from speech representations trained with a large amount of data. The disentangled embeddings will enable us to achieve better reproduction performance for unseen speakers and rhythm transfer conditioned by different speeches.
  arXiv  Detail & Related papers  (2023-04-24T10:15:58Z)
• Residual Information in Deep Speaker Embedding Architectures [4.619541348328938]
  This paper introduces an analysis over six sets of speaker embeddings extracted with some of the most recent and high-performing DNN architectures. The dataset includes 46 speakers uttering the same set of prompts, recorded in either a professional studio or their home environments. The results show that the discriminative power of the analyzed embeddings is very high, yet across all the analyzed architectures, residual information is still present in the representations.
  arXiv  Detail & Related papers  (2023-02-06T12:37:57Z)
• A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units [94.64927912924087]
  Existing systems ignore the correlation between prosody and language content, leading to degradation of naturalness in converted speech. We devise a cascaded modular system leveraging self-supervised discrete speech units as language representation. Experiments show that our system outperforms previous approaches in naturalness, intelligibility, speaker transferability, and prosody transferability.
  arXiv  Detail & Related papers  (2022-11-12T00:54:09Z)
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios [143.47967241972995]
  We develop AdaSpeech 4, a zero-shot adaptive TTS system for high-quality speech synthesis. We model the speaker characteristics systematically to improve the generalization on new speakers. Without any fine-tuning, AdaSpeech 4 achieves better voice quality and similarity than baselines in multiple datasets.
  arXiv  Detail & Related papers  (2022-04-01T13:47:44Z)
• Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings [53.11450530896623]
  This paper presents a streaming speaker-attributed automatic speech recognition (SA-ASR) model that can recognize "who spoke what" Our model is based on token-level serialized output training (t-SOT) which was recently proposed to transcribe multi-talker speech in a streaming fashion. The proposed model achieves substantially better accuracy than a prior streaming model and shows comparable or sometimes even superior results to the state-of-the-art offline SA-ASR model.
  arXiv  Detail & Related papers  (2022-03-30T21:42:00Z)
• Speaker Anonymization with Distribution-Preserving X-Vector Generation for the VoicePrivacy Challenge 2020 [19.420608243033794]
  We present a Distribution-Preserving Voice Anonymization technique, as our submission to the VoicePrivacy Challenge 2020. We show how this approach generates X-vectors that more closely follow the expected intra-similarity distribution of organic speaker X-vectors.
  arXiv  Detail & Related papers  (2020-10-26T09:53:56Z)
• Vector-Quantized Timbre Representation [53.828476137089325]
  This paper targets a more flexible synthesis of an individual timbre by learning an approximate decomposition of its spectral properties with a set of generative features. We introduce an auto-encoder with a discrete latent space that is disentangled from loudness in order to learn a quantized representation of a given timbre distribution. We detail results for translating audio between orchestral instruments and singing voice, as well as transfers from vocal imitations to instruments.
  arXiv  Detail & Related papers  (2020-07-13T12:35:45Z)
• Probabilistic embeddings for speaker diarization [13.276960253126656]
  Speaker embeddings (x-vectors) extracted from very short segments of speech have recently been shown to give competitive performance in speaker diarization. We generalize this recipe by extracting from each speech segment, in parallel with the x-vector, also a diagonal precision matrix. These precisions quantify the uncertainty about what the values of the embeddings might have been if they had been extracted from high quality speech segments.
  arXiv  Detail & Related papers  (2020-04-06T14:51:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.