RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding

• URL: http://arxiv.org/abs/2506.10289v1
• Date: Thu, 12 Jun 2025 02:02:38 GMT
• Title: RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding
• Abstract summary: RT-VC is a zero-shot real-time voice conversion system that delivers ultra-low latency and high-quality performance.<n>We show that RT-VC achieves a CPU latency of 61.4 ms, representing a 13.3% reduction in latency.
• Score: 1.6012334795166434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Voice conversion has emerged as a pivotal technology in numerous applications ranging from assistive communication to entertainment. In this paper, we present RT-VC, a zero-shot real-time voice conversion system that delivers ultra-low latency and high-quality performance. Our approach leverages an articulatory feature space to naturally disentangle content and speaker characteristics, facilitating more robust and interpretable voice transformations. Additionally, the integration of differentiable digital signal processing (DDSP) enables efficient vocoding directly from articulatory features, significantly reducing conversion latency. Experimental evaluations demonstrate that, while maintaining synthesis quality comparable to the current state-of-the-art (SOTA) method, RT-VC achieves a CPU latency of 61.4 ms, representing a 13.3\% reduction in latency.

Related papers

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space [68.11390597559101]
  X-VC is a zero-shot streaming VC system that performs one-step conversion in the latent space of a pretrained neural.<n>X-VC achieves the best streaming WER in both English and Chinese, strong speaker similarity in same-latency and cross-lingual settings.
  arXiv  Detail & Related papers  (2026-04-14T08:42:10Z)
• Voxtral Realtime [134.66962524291424]
  Voxtral Realtime is a streaming automatic speech recognition model.<n>It matches offline transcription quality at sub-second latency.<n>We release the model weights under the Apache 2.0 license.
  arXiv  Detail & Related papers  (2026-02-11T19:17:10Z)
• HQ-SVC: Towards High-Quality Zero-Shot Singing Voice Conversion in Low-Resource Scenarios [18.036712630643205]
  HQ-SVC is an efficient framework for high-quality zero-shot singing voice conversion.<n> HQ-SVC first extracts jointly content and speaker features using a decoupled model.<n>It then enhances fidelity through pitch and volume modeling, preserving critical acoustic information.
  arXiv  Detail & Related papers  (2025-11-11T17:33:30Z)
• MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows [13.130255838403002]
  MeanAudio is a fast and faithful text-to-audio generator capable of rendering realistic sound with only one function evaluation (1-NFE)<n>We demonstrate that MeanAudio achieves state-of-the-art performance in single-step audio generation.
  arXiv  Detail & Related papers  (2025-08-08T07:49:59Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation [21.216297567167036]
  MirrorMe is a real-time, controllable framework built on the LTX video model.<n>MirrorMe compresses video spatially and temporally for efficient latent space denoising.<n> experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability.
  arXiv  Detail & Related papers  (2025-06-27T09:57:23Z)
• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling [50.537794606598254]
  StreamMel is a pioneering single-stage streaming TTS framework that models continuous mel-spectrograms.<n>It enables low-latency, autoregressive synthesis while preserving high speaker similarity and naturalness.<n>It even achieves performance comparable to offline systems while supporting efficient real-time generation.
  arXiv  Detail & Related papers  (2025-06-14T16:53:39Z)
• Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling [76.23539797803681]
  Existing methods primarily use a look mechanism, relying on future text to achieve natural streaming speech synthesis.<n>We propose LE, a streaming framework for generating high-quality speech frame-by-frame.<n> Experimental results suggest that the LE outperforms current streaming TTS methods and achieves comparable performance over sentence-level TTS systems.
  arXiv  Detail & Related papers  (2025-05-26T08:25:01Z)
• Towards Sub-millisecond Latency Real-Time Speech Enhancement Models on Hearables [21.542503235873227]
  Low latency models are critical for real-time speech enhancement applications, such as hearing aids and hearables.<n>We demonstrate speech enhancement using a computationally efficient minimum-phase FIR filter, enabling sample-by-sample processing to achieve mean algorithmic latency of 0.32 ms to 1.25 ms.<n>We hope this work will enable a better understanding of latency and can be used to improve the comfort and usability of hearables.
  arXiv  Detail & Related papers  (2024-09-26T19:31:05Z)
• Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP [6.295981052578859]
  Articulatory trajectories like electromagnetic articulography (EMA) provide a low-dimensional representation of the vocal tract filter. We propose a fast, high-quality, and parameter-efficientP articulatory vocoder that can synthesize speech from EMA, F0, and loudness. Our model achieves a transcription word error rate (WER) of 6.67% and a mean opinion score (MOS) of 3.74, with an improvement of 1.63% and 0.16 compared to the state-of-the-art (SOTA) baseline.
  arXiv  Detail & Related papers  (2024-09-04T05:12:15Z)
• RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement [36.10772098876638]
  We propose RT-LA-VocE, which re-designs every component of LA-VocE to perform causal real-time inference with a 40ms input frame. We show that our algorithm achieves state-of-the-art results in all real-time scenarios.
  arXiv  Detail & Related papers  (2024-07-10T16:49:23Z)
• OFDM-Standard Compatible SC-NOFS Waveforms for Low-Latency and Jitter-Tolerance Industrial IoT Communications [53.398544571833135]
  This work proposes a spectrally efficient irregular Sinc (irSinc) shaping technique, revisiting the traditional Sinc back to 1924. irSinc yields a signal with increased spectral efficiency without sacrificing error performance. Our signal achieves faster data transmission within the same spectral bandwidth through 5G standard signal configuration.
  arXiv  Detail & Related papers  (2024-06-07T09:20:30Z)
• StreamVC: Real-Time Low-Latency Voice Conversion [20.164321451712564]
  StreamVC is a streaming voice conversion solution that preserves the content and prosody of any source speech while matching the voice timbre from any target speech. StreamVC produces the resulting waveform at low latency from the input signal even on a mobile platform.
  arXiv  Detail & Related papers  (2024-01-05T22:37:26Z)
• Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments [49.38965743465124]
  This paper introduces a streaming Transformer-Transducer that jointly generates automatic speech recognition (ASR) and speech translation (ST) outputs using a single decoder. Experiments in monolingual and multilingual settings demonstrate that our approach achieves the best quality-latency balance.
  arXiv  Detail & Related papers  (2023-07-07T02:26:18Z)
• Building Accurate Low Latency ASR for Streaming Voice Search [0.0]
  This work focuses on developing accurate LSTM, attention, and CTC based streaming ASR models for large-scale Voice Search. We investigate various modifications in vanilla LSTM training which enhance the system's accuracy while preserving its streaming capabilities. We present a simple training and inference strategy for end-to-end models that enables joint ASR and EOS detection.
  arXiv  Detail & Related papers  (2023-05-29T20:24:14Z)
• DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion [51.83469048737548]
  We propose DiffSVC, an SVC system based on denoising diffusion probabilistic model. A denoising module is trained in DiffSVC, which takes destroyed mel spectrogram and its corresponding step information as input to predict the added Gaussian noise. Experiments show that DiffSVC can achieve superior conversion performance in terms of naturalness and voice similarity to current state-of-the-art SVC approaches.
  arXiv  Detail & Related papers  (2021-05-28T14:26:40Z)
• Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers [56.56220390953412]
  We extend our prior work by introducing the Conformer architecture to further improve the accuracy. We demonstrate that the extended Transformer provides state-of-the-art end-to-end ASR performance.
  arXiv  Detail & Related papers  (2021-04-19T16:18:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.