Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model

• URL: http://arxiv.org/abs/2607.26742v1
• Date: Wed, 29 Jul 2026 10:33:56 GMT
• Title: Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model
• Abstract summary: We propose a Face-to-Speech framework that predicts a plausible voice from a static facial image.<n>A lightweight Face Adapter aligns face-recognition features with the style space of a frozen StyleTTS 2 model.<n>We evaluate on held-out identities from LRS3, a large-scale audiovisual corpus of English TED-talk videos.
• Score: 0.5729426778193398
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Zero-shot text-to-speech (TTS) clones a voice from a short audio prompt, but this reliance on reference audio is a barrier when only visual information is available, e.g. for historical figures or video-game characters. In this work, we propose a Face-to-Speech (F2S) framework that predicts a plausible voice from a static facial image. A lightweight Face Adapter, together with soft-tuning of the face encoder's upper blocks, aligns face-recognition features with the style space of a frozen StyleTTS 2 model, kept frozen during training. We evaluate on held-out identities from LRS3, a large-scale audiovisual corpus of English TED-talk videos. The synthesized speech is highly natural (UTMOS 3.7-4.0, matching or exceeding the 3.61 of ground truth), face-to-voice retrieval is consistently above chance, and the generated voice is consistent with the target speaker. Without any retraining, an English-trained adapter also produces fluent Spanish speech, indicating that the face-to-style mapping is largely language-agnostic.

Related papers

• Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering [53.2204901422631]
  Text2Lip is a viseme-centric framework that constructs an interpretable phonetic-visual bridge.<n>We show that Text2Lip outperforms existing approaches in semantic fidelity, visual realism, and modality robustness.
  arXiv  Detail & Related papers  (2025-08-04T12:50:22Z)
• Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis [52.25128289155576]
  This paper explores multi-modal controllable Text-to-Speech Synthesis (TTS) where the voice can be generated from face image.<n>We aim to mitigate the following three challenges in face-driven TTS systems.<n> Experimental results validate the proposed model's effectiveness in face-driven voice synthesis.
  arXiv  Detail & Related papers  (2025-05-25T04:43:17Z)
• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training [70.31925012315064]
  We present CosyVoice 3, an improved model designed for zero-shot multilingual speech synthesis in the wild.<n>Key features of CosyVoice 3 include a novel speech tokenizer to improve prosody naturalness.<n>Data is expanded from ten thousand hours to one million hours, encompassing 9 languages and 18 Chinese dialects.
  arXiv  Detail & Related papers  (2025-05-23T07:55:21Z)
• GaussianSpeech: Audio-Driven Gaussian Avatars [76.10163891172192]
  We introduce GaussianSpeech, a novel approach that synthesizes high-fidelity animation sequences of photo-realistic, personalized 3D human head avatars from spoken audio.<n>We propose a compact and efficient 3DGS-based avatar representation that generates expression-dependent color and leverages wrinkle- and perceptually-based losses to synthesize facial details.
  arXiv  Detail & Related papers  (2024-11-27T18:54:08Z)
• GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance [83.43852715997596]
  GSmoothFace is a novel two-stage generalized talking face generation model guided by a fine-grained 3d face model. It can synthesize smooth lip dynamics while preserving the speaker's identity. Both quantitative and qualitative experiments confirm the superiority of our method in terms of realism, lip synchronization, and visual quality.
  arXiv  Detail & Related papers  (2023-12-12T16:00:55Z)
• Face-StyleSpeech: Enhancing Zero-shot Speech Synthesis from Face Images with Improved Face-to-Speech Mapping [37.57813713418656]
  We propose Face-StyleSpeech, a zero-shot Text-To-Speech model that generates natural speech conditioned on a face image.<n>We show that Face-StyleSpeech effectively generates more natural speech from a face image than baselines, even for unseen faces.
  arXiv  Detail & Related papers  (2023-09-25T13:46:00Z)
• Zero-shot personalized lip-to-speech synthesis with face image based voice control [41.17483247506426]
  Lip-to-Speech (Lip2Speech) synthesis, which predicts corresponding speech from talking face images, has witnessed significant progress with various models and training strategies. We propose a zero-shot personalized Lip2Speech synthesis method, in which face images control speaker identities.
  arXiv  Detail & Related papers  (2023-05-09T02:37:29Z)
• Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech [33.01930038988336]
  We introduce a face-styled diffusion text-to-speech (TTS) model within a unified framework, called Face-TTS. We jointly train cross-model biometrics and TTS models to preserve speaker identity between face images and generated speech segments. Since the biometric information is extracted directly from the face image, our method does not require extra fine-tuning steps to generate speech from unseen and unheard speakers.
  arXiv  Detail & Related papers  (2023-02-27T11:59:28Z)
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis [68.42632589736881]
  This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice. GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components. Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity.
  arXiv  Detail & Related papers  (2022-05-15T08:16:02Z)
• AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person [21.126759304401627]
  We present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input. Experiments demonstrate that the proposed method is able to generate synchronized speech and talking head videos for arbitrary persons and non-persons.
  arXiv  Detail & Related papers  (2021-08-09T19:58:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.