TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation

• URL: http://arxiv.org/abs/2603.06057v1
• Date: Fri, 06 Mar 2026 09:09:01 GMT
• Title: TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation
• Abstract summary: This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework.<n>It explores few-step inference for efficient audio-driven talking-head generation.<n>The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Diffusion models have recently advanced photorealistic human synthesis, although practical talking-head generation (THG) remains constrained by high inference latency, temporal instability such as flicker and identity drift, and imperfect audio-visual alignment under challenging speech conditions. This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework that explores few-step inference for efficient audio-driven talking-head generation. The approach adopts a teacher-student distillation formulation in which a diffusion teacher trained with a standard noise prediction objective guides a lightweight student denoiser capable of operating with significantly fewer inference steps to improve generation stability. The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker during synthesis, while viseme-based audio conditioning provides coarse lip motion control. Experiments on the LRS3 dataset report denoising-stage component-level metrics relative to VAE reconstructions and preliminary latency characterization, including CPU-only and edge computing measurements and feasibility estimates for edge deployment. The results suggest that distilled diffusion models can retain much of the reconstruction behaviour of a stronger teacher while enabling substantially lower latency inference. The study is positioned as an initial step toward practical diffusion-based talking-head generation under constrained computational settings. GitHub: https://mazumdarsoumya.github.io/TempoSyncDiff

Related papers

• CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents [53.73961877379327]
  CuteTTS is a compact continuous-autoregressive TTS system.<n>It combines semantically aligned causal VAE latents with patch-level autoregression, explicit speaker conditioning, and a bidirectional flow-matching head.
  arXiv  Detail & Related papers  (2026-08-09T11:10:20Z)
• LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation [28.818825505825476]
  We propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step.<n>Our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS.
  arXiv  Detail & Related papers  (2026-07-29T14:44:57Z)
• Probing Diffusion Denoising Dynamics for Contrastive Representation Learning [48.679227622825806]
  We study how can the denoising dynamics of a pretrained diffusion model be adapted to support discriminative representation learning.<n>We present D$3$CL as an investigation of this question.<n>We show that noise-level contrastive learning can structure denoising representations for discriminative tasks while maintaining generative performance.
  arXiv  Detail & Related papers  (2026-07-10T03:23:44Z)
• Test-Time Self-Adaptive Conditioning for Stable Audio-Driven Talking-Head Generation [28.732587811107777]
  We introduce Test-Time Self-Adaptive Conditioning (TT-SAC), a parameter-free inference framework that enables pretrained talking-head generators to adapt their conditioning representations during inference without retraining, gradient updates, or additional supervision.<n>A single adaptation step approximates a self-consistent equilibrium of the generative process, stabilizing identity and motion across time.<n>Experiments on state-of-the-art talking-head generators and benchmark datasets demonstrate consistent improvements in lip-sync accuracy, temporal coherence, identity preservation, and perceptual fidelity.
  arXiv  Detail & Related papers  (2026-05-25T06:45:29Z)
• SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning [17.14027883239169]
  We propose a post-training framework SyncDPO to improve the temporal sensitivity of V-A joint generation.<n>We demonstrate that SyncDPO significantly outperforms other methods in improving model's temporal alignment capability.
  arXiv  Detail & Related papers  (2026-05-12T14:22:13Z)
• Enhancing Consistency Models for Multi-Agent Trajectory Prediction [9.313920952984104]
  ECTraj is an enhanced CM pipeline with improved training and conditional generation for trajectory prediction.<n>We exploit CMs' direct denoising for top-K multi-shot generation during training.
  arXiv  Detail & Related papers  (2026-05-09T00:23:08Z)
• REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation [41.34425148954312]
  REST bridges the gap between autoregressive and diffusion-based approaches for talking head generation.<n>We show that REST outperforms state-of-the-art methods in both generation speed and overall performance.
  arXiv  Detail & Related papers  (2025-12-12T02:28:52Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning [79.44594332189018]
  Class-Incremental Learning (CIL) seeks to develop models that continuously learn new action categories over time without previously acquired knowledge.<n>Existing approaches either rely on forgetting, raising concerns over memory and privacy, or adapt static image-based methods that neglect temporal modeling.<n>We propose a unified and exemplar-free VCIL framework that explicitly disentangles and preserves information.
  arXiv  Detail & Related papers  (2025-05-20T06:46:51Z)
• ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation [21.335983674309475]
  Diffusion models suffer from slow inference due to an excessive number of queries to the underlying denoising network per generation. We introduce ConsistencyTTA, a framework requiring only a single non-autoregressive network query. We achieve so by proposing "CFG-aware latent consistency model," which adapts consistency generation into a latent space.
  arXiv  Detail & Related papers  (2023-09-19T16:36:33Z)
• Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learning [71.24316734338501]
  We propose an effective temporally-conditional diffusion model coined Temporally-Composable diffuser (TCD) TCD extracts temporal information from interaction sequences and explicitly guides generation with temporal conditions. Our method reaches or matches the best performance compared with prior SOTA baselines.
  arXiv  Detail & Related papers  (2023-06-08T02:12:26Z)
• Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation [66.46123655365113]
  Target speech separation refers to extracting the target speaker's speech from mixed signals. Two main challenges are the complex acoustic environment and the real-time processing requirement. We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
  arXiv  Detail & Related papers  (2020-01-02T11:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.