Provable Speech Attributes Conversion via Latent Independence

• URL: http://arxiv.org/abs/2510.05191v2
• Date: Thu, 09 Oct 2025 08:32:27 GMT
• Title: Provable Speech Attributes Conversion via Latent Independence
• Authors: Jonathan Svirsky, Ofir Lindenbaum, Uri Shaham,
• Abstract summary: We propose a general framework for speech attribute conversion, accompanied by theoretical analysis and guarantees under reasonable assumptions.<n>Our framework builds on a non-probabilistic autoencoder architecture with an independence constraint between the predicted latent variable and the target controllable variable.<n>This design ensures a consistent signal transformation, conditioned on an observed style variable, while preserving the original content and modifying the desired attribute.
• Score: 22.02196595272211
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While signal conversion and disentangled representation learning have shown promise for manipulating data attributes across domains such as audio, image, and multimodal generation, existing approaches, especially for speech style conversion, are largely empirical and lack rigorous theoretical foundations to guarantee reliable and interpretable control. In this work, we propose a general framework for speech attribute conversion, accompanied by theoretical analysis and guarantees under reasonable assumptions. Our framework builds on a non-probabilistic autoencoder architecture with an independence constraint between the predicted latent variable and the target controllable variable. This design ensures a consistent signal transformation, conditioned on an observed style variable, while preserving the original content and modifying the desired attribute. We further demonstrate the versatility of our method by evaluating it on speech styles, including speaker identity and emotion. Quantitative evaluations confirm the effectiveness and generality of the proposed approach.

Related papers

• Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation [57.97548022208733]
  We show that seemingly superficial choices in key-value extraction can induce shifts in accuracy and stability.<n>We introduce Contextual Normalization, a strategy that adaptively standardizes context representations before generation.
  arXiv  Detail & Related papers  (2025-10-15T06:28:25Z)
• Unified modality separation: A vision-language framework for unsupervised domain adaptation [60.8391821117794]
  Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains.<n>We propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components.<n>Our methods achieve up to 9% performance gain with 9 times of computational efficiencies.
  arXiv  Detail & Related papers  (2025-08-07T02:51:10Z)
• AGENT-X: Adaptive Guideline-based Expert Network for Threshold-free AI-generated teXt detection [44.66668435489055]
  AGENT-X is a zero-shot multi-agent framework for AI-generated text detection.<n>We organize detection guidelines into semantic, stylistic, and structural dimensions, each independently evaluated by specialized linguistic agents.<n>A meta agent integrates these assessments through confidence-aware aggregation, enabling threshold-free, interpretable classification.<n>Experiments on diverse datasets demonstrate that AGENT-X substantially surpasses state-of-the-art supervised and zero-shot approaches in accuracy, interpretability, and generalization.
  arXiv  Detail & Related papers  (2025-05-21T08:39:18Z)
• Independence Constrained Disentangled Representation Learning from Epistemological Perspective [13.51102815877287]
  Disentangled Representation Learning aims to improve the explainability of deep learning methods by training a data encoder that identifies semantically meaningful latent variables in the data generation process. There is no consensus regarding the objective of disentangled representation learning. We propose a novel method for disentangled representation learning by employing an integration of mutual information constraint and independence constraint.
  arXiv  Detail & Related papers  (2024-09-04T13:00:59Z)
• STAB: Speech Tokenizer Assessment Benchmark [57.45234921100835]
  Representing speech as discrete tokens provides a framework for transforming speech into a format that closely resembles text. We present STAB (Speech Tokenizer Assessment Benchmark), a systematic evaluation framework designed to assess speech tokenizers comprehensively. We evaluate the STAB metrics and correlate this with downstream task performance across a range of speech tasks and tokenizer choices.
  arXiv  Detail & Related papers  (2024-09-04T02:20:59Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders [0.0]
  We argue that continuous variables may not be ideal to model features of textual data, due to the fact that most generative factors in text are discrete. We propose a Variational Autoencoder based method which models language features as discrete variables and encourages independence between variables for learning disentangled representations.
  arXiv  Detail & Related papers  (2021-09-15T09:10:05Z)
• Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis [19.386519810463003]
  This paper proposes a hierarchical generative model with a multi-grained latent variable to synthesize expressive speech. Our proposed framework also provides the controllability of speaking style in an entire utterance.
  arXiv  Detail & Related papers  (2020-09-17T18:00:19Z)
• Nonlinear ISA with Auxiliary Variables for Learning Speech Representations [51.9516685516144]
  We introduce a theoretical framework for nonlinear Independent Subspace Analysis (ISA) in the presence of auxiliary variables. We propose an algorithm that learns unsupervised speech representations whose subspaces are independent.
  arXiv  Detail & Related papers  (2020-07-25T14:53:09Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Unsupervised Representation Disentanglement using Cross Domain Features and Adversarial Learning in Variational Autoencoder based Voice Conversion [28.085498706505774]
  An effective approach for voice conversion (VC) is to disentangle linguistic content from other components in the speech signal. In this paper, we extend the CDVAE-VC framework by incorporating the concept of adversarial learning.
  arXiv  Detail & Related papers  (2020-01-22T02:06:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.