ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search

• URL: http://arxiv.org/abs/2511.06833v1
• Date: Mon, 10 Nov 2025 08:28:13 GMT
• Title: ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search
• Authors: Zhenjie Liu, Jianzhang Lu, Renjie Lu, Cong Liang, Shangfei Wang,
• Abstract summary: We introduce textbfTalk, a novel intensity-controllable talking head generation framework with diffusion noise search inference.<n>First, we propose textbfan optical flow-guided temporal module (OFT) that decouples motion features from static appearance.<n>Second, we present an textbfAudio-to-Intensity (A2I) model obtained through multimodal teacher-student knowledge distillation.
• Score: 8.993664585683055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in video diffusion models have significantly enhanced audio-driven portrait animation. However, current methods still suffer from flickering, identity drift, and poor audio-visual synchronization. These issues primarily stem from entangled appearance-motion representations and unstable inference strategies. In this paper, we introduce \textbf{ConsistTalk}, a novel intensity-controllable and temporally consistent talking head generation framework with diffusion noise search inference. First, we propose \textbf{an optical flow-guided temporal module (OFT)} that decouples motion features from static appearance by leveraging facial optical flow, thereby reducing visual flicker and improving temporal consistency. Second, we present an \textbf{Audio-to-Intensity (A2I) model} obtained through multimodal teacher-student knowledge distillation. By transforming audio and facial velocity features into a frame-wise intensity sequence, the A2I model enables joint modeling of audio and visual motion, resulting in more natural dynamics. This further enables fine-grained, frame-wise control of motion dynamics while maintaining tight audio-visual synchronization. Third, we introduce a \textbf{diffusion noise initialization strategy (IC-Init)}. By enforcing explicit constraints on background coherence and motion continuity during inference-time noise search, we achieve better identity preservation and refine motion dynamics compared to the current autoregressive strategy. Extensive experiments demonstrate that ConsistTalk significantly outperforms prior methods in reducing flicker, preserving identity, and delivering temporally stable, high-fidelity talking head videos.

Related papers

• Adaptive Begin-of-Video Tokens for Autoregressive Video Diffusion Models [11.913945404405865]
  Most video diffusion models (VDMs) generate videos in an autoregressive manner, generating subsequent iteration frames conditioned on previous ones.<n>We propose Adaptive Begin-of-Video Tokens (ada-BOV) for autoregressive VDMs.
  arXiv  Detail & Related papers  (2025-11-15T08:29:14Z)
• DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis [15.304037069236536]
  DEMO is a flow-matching generative framework for audio-driven talking-head video synthesis.<n>It delivers disentangled, high-fidelity control of lip motion, head pose, and eye gaze.
  arXiv  Detail & Related papers  (2025-10-12T15:10:33Z)
• StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbing [63.72095377128904]
  The visual dubbing task aims to generate mouth movements synchronized with the driving audio.<n>Audio-only driving paradigms inadequately capture speaker-specific lip habits.<n>Blind-inpainting approaches produce visual artifacts when handling obstructions.
  arXiv  Detail & Related papers  (2025-09-26T05:23:31Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation [65.48046909056468]
  We reformulate talking head generation into a unified framework comprising video preprocessing, motion representation, and rendering reconstruction.<n>M2DAO-Talker achieves state-of-the-art performance, with the 2.43 dB PSNR improvement in generation quality and 0.64 gain in user-evaluated video realness.
  arXiv  Detail & Related papers  (2025-07-11T04:48:12Z)
• MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation [21.216297567167036]
  MirrorMe is a real-time, controllable framework built on the LTX video model.<n>MirrorMe compresses video spatially and temporally for efficient latent space denoising.<n> experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability.
  arXiv  Detail & Related papers  (2025-06-27T09:57:23Z)
• MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation [55.95148886437854]
  Memory-guided EMOtion-aware diffusion (MEMO) is an end-to-end audio-driven portrait animation approach to generate talking videos.<n>MEMO generates more realistic talking videos across diverse image and audio types, outperforming state-of-the-art methods in overall quality, audio-lip synchronization, identity consistency, and expression-emotion alignment.
  arXiv  Detail & Related papers  (2024-12-05T18:57:26Z)
• Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation [36.098738197088124]
  This work presents a Diffusion Reuse MOtion network to accelerate latent video generation. coarse-grained noises in earlier denoising steps have demonstrated high motion consistency across consecutive video frames. Dr. Mo propagates those coarse-grained noises onto the next frame by incorporating carefully designed, lightweight inter-frame motions.
  arXiv  Detail & Related papers  (2024-09-19T07:50:34Z)
• Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency [15.841490425454344]
  We propose an end-to-end audio-only conditioned video diffusion model named Loopy.<n>Specifically, we designed an inter- and intra-clip temporal module and an audio-to-latents module, enabling the model to leverage long-term motion information.
  arXiv  Detail & Related papers  (2024-09-04T11:55:14Z)
• Live2Diff: Live Stream Translation via Uni-directional Attention in Video Diffusion Models [64.2445487645478]
  Large Language Models have shown remarkable efficacy in generating streaming data such as text and audio. We present Live2Diff, the first attempt at designing a video diffusion model with uni-directional temporal attention, specifically targeting live streaming video translation.
  arXiv  Detail & Related papers  (2024-07-11T17:34:51Z)
• TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models [94.24861019513462]
  TRIP is a new recipe of image-to-video diffusion paradigm. It pivots on image noise prior derived from static image to jointly trigger inter-frame relational reasoning. Extensive experiments on WebVid-10M, DTDB and MSR-VTT datasets demonstrate TRIP's effectiveness.
  arXiv  Detail & Related papers  (2024-03-25T17:59:40Z)
• Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation [41.292644854306594]
  We propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture) DiffGesture achieves state-of-theart performance, which renders coherent gestures with better mode coverage and stronger audio correlations.
  arXiv  Detail & Related papers  (2023-03-16T07:32:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.