Test-Time Self-Adaptive Conditioning for Stable Audio-Driven Talking-Head Generation

• URL: http://arxiv.org/abs/2605.25488v1
• Date: Mon, 25 May 2026 06:45:29 GMT
• Title: Test-Time Self-Adaptive Conditioning for Stable Audio-Driven Talking-Head Generation
• Abstract summary: We introduce Test-Time Self-Adaptive Conditioning (TT-SAC), a parameter-free inference framework that enables pretrained talking-head generators to adapt their conditioning representations during inference without retraining, gradient updates, or additional supervision.<n>A single adaptation step approximates a self-consistent equilibrium of the generative process, stabilizing identity and motion across time.<n>Experiments on state-of-the-art talking-head generators and benchmark datasets demonstrate consistent improvements in lip-sync accuracy, temporal coherence, identity preservation, and perceptual fidelity.
• Score: 28.732587811107777
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-driven talking-head generation has achieved remarkable progress with recent models such as AniTalker, FLOAT, and Sonic. Despite their success, most existing approaches rely on a single static reference image to condition the entire video generation process at inference stage. This static conditioning paradigm often creates a mismatch between fixed identity features and dynamically evolving facial motion, leading to identity drift, temporal inconsistency, and degraded perceptual quality. We introduce Test-Time Self-Adaptive Conditioning (TT-SAC), a parameter-free inference framework that enables pretrained talking-head generators to adapt their conditioning representations during inference without retraining, gradient updates, or additional supervision. Instead of treating the reference portrait as immutable, TT-SAC composes the generator with its encoder in a feedback loop: the generator's own outputs are re-encoded to construct a refined conditioning representation that better aligns with the temporal dynamics of the synthesized sequence. A single adaptation step approximates a self-consistent equilibrium of the generative process, stabilizing identity and motion across time. We further provide theoretical analysis showing that test-time conditioning adaptation reduces feature variance and improves generative stability under mild Lipschitz assumptions, while exhibiting a principled bias-variance tradeoff that governs the optimal strength of adaptation. Extensive experiments on state-of-the-art talking-head generators and benchmark datasets demonstrate consistent improvements in lip-sync accuracy, temporal coherence, identity preservation, and perceptual fidelity. TT-SAC offers a model-agnostic and training-free strategy for enhancing generative video models, establishing test-time conditioning adaptation as an effective mechanism for stabilizing audio-driven portrait animation.

Related papers

• Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition [50.15288513086986]
  Contrastive decoding (CD) originally introduced to stabilize large language model (LLM)-based audio-visual speech recognition (AVSR) systems.<n>We propose reliability-aware scaling of CD for AVSR.
  arXiv  Detail & Related papers  (2026-08-26T08:25:15Z)
• Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition [45.191945465591374]
  Task-specific lightweight models for surgical phase recognition excel at capturing temporal dynamics but generalize poorly under domain shift.<n>We propose textbfLarge-textbfSmall textbfTemporal adaptation (textbfLaST), a novel large-small collaborative framework that enables zero-shot adaptation to unseen clinical domains.
  arXiv  Detail & Related papers  (2026-08-24T07:06:52Z)
• Equilibrium Forcing: Adaptive Video Generation Without Noise Conditioning [85.93126001283169]
  We introduce Equilibrium Forcing (EqF), a framework for video denoising generative models without noise level conditioning.<n>This flexibility allows for inference-time algorithms that operate in a closed loop by adapting to feedback from the sample.
  arXiv  Detail & Related papers  (2026-08-11T00:02:32Z)
• CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents [53.73961877379327]
  CuteTTS is a compact continuous-autoregressive TTS system.<n>It combines semantically aligned causal VAE latents with patch-level autoregression, explicit speaker conditioning, and a bidirectional flow-matching head.
  arXiv  Detail & Related papers  (2026-08-09T11:10:20Z)
• Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation [28.732587811107777]
  Existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos.<n>We introduce a unified sequence-level reformulation that integrates Soft Dynamic Time Warping into established evaluation pipelines.<n>We show that frame-wise evaluation can be viewed as a special case under rigid alignment, while sequence-level alignment provides improved stability, lower sensitivity to timing differences, and clearer separation between modeling paradigms.
  arXiv  Detail & Related papers  (2026-05-31T05:44:42Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation [0.0]
  This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework.<n>It explores few-step inference for efficient audio-driven talking-head generation.<n>The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker.
  arXiv  Detail & Related papers  (2026-03-06T09:09:01Z)
• Knob: A Physics-Inspired Gating Interface for Interpretable and Controllable Neural Dynamics [7.965536008626047]
  Knob is a framework that connects deep learning with classical control theory.<n>Our framework allows operators to tune "stability" and "sensitivity" through familiar physical analogues.
  arXiv  Detail & Related papers  (2026-02-26T07:25:22Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• Generative Pre-trained Autoregressive Diffusion Transformer [74.25668109048418]
  GPDiT is a Generative Pre-trained Autoregressive Diffusion Transformer.<n>It unifies the strengths of diffusion and autoregressive modeling for long-range video synthesis.<n>It autoregressively predicts future latent frames using a diffusion loss, enabling natural modeling of motion dynamics.
  arXiv  Detail & Related papers  (2025-05-12T08:32:39Z)
• TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis [39.58643124601554]
  This paper introduces Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning (TARO), a novel framework for high-fidelity and temporally coherent video-to-audio synthesis.
  arXiv  Detail & Related papers  (2025-04-08T04:49:36Z)
• Advancing Test-Time Adaptation in Wild Acoustic Test Settings [26.05732574338255]
  Speech signals follow short-term consistency, requiring specialized adaptation strategies. We propose a novel wild acoustic TTA method tailored for ASR fine-tuned acoustic foundation models. Our approach outperforms existing baselines under various wild acoustic test settings.
  arXiv  Detail & Related papers  (2023-10-14T06:22:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.