X-VC: Zero-shot Streaming Voice Conversion in Codec Space

• URL: http://arxiv.org/abs/2604.12456v2
• Date: Wed, 22 Apr 2026 07:37:41 GMT
• Title: X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Abstract summary: X-VC is a zero-shot streaming VC system that performs one-step conversion in the latent space of a pretrained neural.<n>X-VC achieves the best streaming WER in both English and Chinese, strong speaker similarity in same-latency and cross-lingual settings.
• Score: 68.11390597559101
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Zero-shot voice conversion (VC) aims to convert a source utterance into the voice of an unseen target speaker while preserving its linguistic content. Although recent systems have improved conversion quality, building zero-shot VC systems for interactive scenarios remains challenging because high-fidelity speaker transfer and low-latency streaming inference are difficult to achieve simultaneously. In this work, we present X-VC, a zero-shot streaming VC system that performs one-step conversion in the latent space of a pretrained neural codec. X-VC uses a dual-conditioning acoustic converter that jointly models source codec latents and frame-level acoustic conditions derived from target reference speech, while injecting utterance-level target speaker information through adaptive normalization. To reduce the mismatch between training and inference, we train the model with generated paired data and a role-assignment strategy that combines standard, reconstruction, and reversed modes. For streaming inference, we further adopt a chunkwise inference scheme with overlap smoothing that is aligned with the segment-based training paradigm of the codec. Experiments on Seed-TTS-Eval show that X-VC achieves the best streaming WER in both English and Chinese, strong speaker similarity in same-language and cross-lingual settings, and substantially lower offline real-time factor than the compared baselines. These results suggest that codec-space one-step conversion is a practical approach for building high-quality low-latency zero-shot VC systems. Our audio samples, code and checkpoints are released at https://github.com/Jerrister/X-VC.

Related papers

• Hierarchical Codec Diffusion for Video-to-Speech Generation [34.08427878034203]
  Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals.<n>Existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details.<n>We propose HiCoDiT, a novel Hierarchical Codec Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment.
  arXiv  Detail & Related papers  (2026-04-17T10:28:21Z)
• SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec [83.61175662066364]
  Speech codecs serve as a crucial bridge in unifying speech and text language models.<n>Existing methods face several challenges in semantic encoding.<n>We propose SecoustiCodec, a cross-modal aligned low-bitrate streaming speech codecs.
  arXiv  Detail & Related papers  (2025-08-04T19:22:14Z)
• FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks [33.022035588157614]
  FocalCodec is an efficient low-bitrate based on focal modulation that utilizes a single binary codebook to compress speech.<n>Demo samples and code are available at https://lucadellalib.io/kbpscodec-web/.
  arXiv  Detail & Related papers  (2025-02-06T19:24:50Z)
• AdaptVC: High Quality Voice Conversion with Adaptive Learning [28.25726543043742]
  Key challenge is to extract disentangled linguistic content from the source and voice style from the reference.<n>In this paper, we achieve successful disentanglement of content and speaker features by tuning self-supervised speech features with adapters.<n>The adapters are trained to dynamically encode nuanced features from rich self-supervised features, and the decoder fuses them to produce speech that accurately resembles the reference.
  arXiv  Detail & Related papers  (2025-01-02T16:54:08Z)
• Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling [14.98368067290024]
  Takin-VC is a novel expressive zero-shot voice conversion framework.<n>We introduce an innovative hybrid content encoder that incorporates an adaptive fusion module.<n>For timbre modeling, we propose advanced memory-augmented and context-aware modules.
  arXiv  Detail & Related papers  (2024-10-02T09:07:33Z)
• VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion [77.50171525265056]
  This paper proposes a novel multi-speaker Video-to-Speech (VTS) system based on cross-modal knowledge transfer from voice conversion (VC) The Lip2Ind network can substitute the content encoder of VC to form a multi-speaker VTS system to convert silent video to acoustic units for reconstructing accurate spoken content.
  arXiv  Detail & Related papers  (2022-02-18T08:58:45Z)
• StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion [19.74933410443264]
  We present an unsupervised many-to-many voice conversion (VC) method using a generative adversarial network (GAN) called StarGAN v2. Our model is trained only with 20 English speakers. It generalizes to a variety of voice conversion tasks, such as any-to-many, cross-lingual, and singing conversion.
  arXiv  Detail & Related papers  (2021-07-21T23:44:17Z)
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion [54.29557210925752]
  One-shot voice conversion can be effectively achieved by speech representation disentanglement. We employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations.
  arXiv  Detail & Related papers  (2021-06-18T13:50:38Z)
• Voicy: Zero-Shot Non-Parallel Voice Conversion in Noisy Reverberant Environments [76.98764900754111]
  Voice Conversion (VC) is a technique that aims to transform the non-linguistic information of a source utterance to change the perceived identity of the speaker. We propose Voicy, a new VC framework particularly tailored for noisy speech. Our method, which is inspired by the de-noising auto-encoders framework, is comprised of four encoders (speaker, content, phonetic and acoustic-ASR) and one decoder.
  arXiv  Detail & Related papers  (2021-06-16T15:47:06Z)
• StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contexts [32.170748231414365]
  To be useful in a wider range of contexts, voice conversion systems need to be trainable without access to parallel data. This paper extends recent voice conversion models based on generative adversarial networks (GANs) We show that real-time zero-shot voice conversion is possible even for a model trained on very little data.
  arXiv  Detail & Related papers  (2021-05-31T18:21:28Z)
• NoiseVC: Towards High Quality Zero-Shot Voice Conversion [2.3224617218247126]
  NoiseVC is an approach that can disentangle contents based on VQ and Contrastive Predictive Coding (CPC) We conduct several experiments and demonstrate that NoiseVC has a strong disentanglement ability with a small sacrifice of quality.
  arXiv  Detail & Related papers  (2021-04-13T10:12:38Z)
• VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture [71.45920122349628]
  Auto-encoder-based VC methods disentangle the speaker and the content in input speech without given the speaker's identity. We use the U-Net architecture within an auto-encoder-based VC system to improve audio quality.
  arXiv  Detail & Related papers  (2020-06-07T14:01:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.