Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

• URL: http://arxiv.org/abs/2608.15690v1
• Date: Sun, 16 Aug 2026 11:40:20 GMT
• Title: Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
• Abstract summary: We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-d linear layer on top of its audio backbone.<n>A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a 30x speed-up over the full audio-video diffusion loop.
• Score: 1.5678079269619796
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.

Related papers

• TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context [0.04583541422554719]
  TontaubeV1 is a model that preserves natural prosody while enabling streaming from a single consumer GPU.<n>Our design assumes that prosodic structure is largely established when the semantic stream is generated.<n>The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German.
  arXiv  Detail & Related papers  (2026-09-08T13:03:46Z)
• Unified Audio Intelligence Without Regressing on Text Intelligence [86.55418188552768]
  We introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM.<n>Audex adopts a simple unified design with a single Transformer decoder.<n>Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation.
  arXiv  Detail & Related papers  (2026-07-06T15:11:57Z)
• TalkVerse: Democratizing Minute-Long Audio-Driven Video Generation [45.08904139559434]
  We introduce TalkVerse, a large-scale, open corpus for audio-driven talking video generation.<n>TalkVerse offers 2.3 million high-resolution (720p/1080p) audio-video synchronized clips totaling 6.3k hours.<n>These are curated from over 60k hours of video via a transparent pipeline.
  arXiv  Detail & Related papers  (2025-12-16T22:01:08Z)
• SoundReactor: Frame-level Online Video-to-Audio Generation [39.113214321291586]
  Video-to-Audio generation models operate offline, assuming an entire video sequence or chunks of frames are available beforehand.<n>We introduce the novel task of frame-level online V2A generation, where a model autoregressively generates audio from video without access to future video frames.<n>SoundReactor is the first simple yet effective framework explicitly tailored for this task.
  arXiv  Detail & Related papers  (2025-10-02T15:18:00Z)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [51.70360630470263]
  Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video.<n>We propose Frieren, a V2A model based on rectified flow matching.<n>Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment.
  arXiv  Detail & Related papers  (2024-06-01T06:40:22Z)
• Text-to-Audio Generation Synchronized with Videos [44.848393652233796]
  We introduce a groundbreaking benchmark for Text-to-Audio generation that aligns with Videos, named T2AV-Bench. We also present a simple yet effective video-aligned TTA generation model, namely T2AV. It employs a temporal multi-head attention transformer to extract and understand temporal nuances from video data, a feat amplified by our Audio-Visual ControlNet.
  arXiv  Detail & Related papers  (2024-03-08T22:27:38Z)
• Large-scale unsupervised audio pre-training for video-to-speech synthesis [64.86087257004883]
  Video-to-speech synthesis is the task of reconstructing the speech signal from a silent video of a speaker. In this paper we propose to train encoder-decoder models on more than 3,500 hours of audio data at 24kHz. We then use the pre-trained decoders to initialize the audio decoders for the video-to-speech synthesis task.
  arXiv  Detail & Related papers  (2023-06-27T13:31:33Z)
• CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models [50.42886595228255]
  We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining model.
  arXiv  Detail & Related papers  (2023-06-16T05:42:01Z)
• SoundStorm: Efficient Parallel Audio Generation [27.121920017380273]
  We present SoundStorm, a model for efficient, non-autoregressive audio generation. SoundStorm receives as semantic tokens of AudioLM, and relies on bidirectional attention and confidence-based parallel decoding. We demonstrate the ability of our model to scale audio generation to longer sequences by synthesizing high-quality, natural dialogue segments.
  arXiv  Detail & Related papers  (2023-05-16T17:41:25Z)
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models [65.18102159618631]
  multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. We propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps.
  arXiv  Detail & Related papers  (2023-01-30T04:44:34Z)
• End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks [54.43697805589634]
  We propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) Our model consists of an encoder-decoder architecture that receives raw video as input and generates speech. We show that this model is able to reconstruct speech with remarkable realism for constrained datasets such as GRID.
  arXiv  Detail & Related papers  (2021-04-27T17:12:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.