Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval

• URL: http://arxiv.org/abs/2604.23692v1
• Date: Sun, 26 Apr 2026 13:15:08 GMT
• Title: Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval
• Abstract summary: We present an end-to-end causal framework for personalizing causal facial motion generation.<n>Our method significantly outperforms state-of-the-art approaches in lip-sync accuracy, identity consistency, and perceived realism.
• Score: 16.85147888618743
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio-driven facial animation is essential for immersive digital interaction, yet existing frameworks fail to reconcile real-time streaming with high-fidelity personalization. Current methods often rely on latency-inducing audio look-ahead, or require high user compliance to pre-encode static embeddings that fails to capture dynamic idiosyncrasies. We present an end-to-end causal framework for personalizing causal facial motion generation via dynamic multi-modal style retrieval, enabling ultra-low latency while uniquely leveraging unstructured style references. We introduce two key innovations: (1) a temporal hierarchical motion representation that captures global temporal context and high-frequency details while maintaining decoding causality, and (2) a multi-modal style retriever that jointly queries audio and motion to dynamically extract stylistic priors without breaking causality. This mechanism allows for scalable personalization with total flexibility regarding the number and contents of templates. By integrating these components into a causal autoregressive architecture, our method significantly outperforms state-of-the-art approaches in lip-sync accuracy, identity consistency, and perceived realism, supported by extensive quantitative evaluations and user studies.

Related papers

• Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation [7.715774279707922]
  We propose a novel implicit-motion framework for generating realistic and temporally coherent human motion videos from a single static image and audio.<n>Our approach uses a two-stage pipeline that decouples motion prediction from rendering.
  arXiv  Detail & Related papers  (2026-06-02T09:43:55Z)
• DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing [27.626489251186968]
  We introduce DiscoForcing, a streaming audio-driven diffusion framework.<n>It captures rhythmic structure and phase dynamics with a diffusion-forcing sequence model trained under heterogeneous noise levels.<n>It delivers more stable long-horizon rollouts and sharper audio-motion alignment than prior baselines under matched causality and latency constraints.
  arXiv  Detail & Related papers  (2026-05-27T13:47:40Z)
• UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars [25.55654497627044]
  UMo is a unified sparse motion modeling architecture for real-time co-speech avatars.<n>It efficiently performs real-time dense reconstruction, enabling temporally coherent and high-fidelity animation generation.<n>It preserves fine-grained speech-motion alignment even under strict latency constraints.
  arXiv  Detail & Related papers  (2026-05-14T11:56:03Z)
• Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation [16.692450893925148]
  We present a novel streaming framework named Knot Forcing for real-time portrait animation.<n>K Knot Forcing enables high-fidelity, temporally consistent, and interactive portrait animation over infinite sequences.
  arXiv  Detail & Related papers  (2025-12-25T16:34:56Z)
• SteadyDancer: Harmonized and Coherent Human Image Animation with First-Frame Preservation [50.792027578906804]
  We introduce SteadyDancer, an Image-to-Video (R2V) paradigm-based framework that achieves harmonized and coherent animation.<n> Experiments demonstrate that SteadyDancer achieves state-of-the-art performance in both appearance fidelity and motion control.
  arXiv  Detail & Related papers  (2025-11-24T17:15:55Z)
• ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search [8.993664585683055]
  We introduce textbfTalk, a novel intensity-controllable talking head generation framework with diffusion noise search inference.<n>First, we propose textbfan optical flow-guided temporal module (OFT) that decouples motion features from static appearance.<n>Second, we present an textbfAudio-to-Intensity (A2I) model obtained through multimodal teacher-student knowledge distillation.
  arXiv  Detail & Related papers  (2025-11-10T08:28:13Z)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing [66.48064661467781]
  We introduce sparse-frame video dubbing, a novel paradigm that strategically preserves references to maintain identity, iconic gestures, and camera trajectories.<n>We propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing.<n> Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-19T17:55:23Z)
• Identity-Preserving Text-to-Video Generation Guided by Simple yet Effective Spatial-Temporal Decoupled Representations [131.33758144860988]
  Identity-preserving text-to-video (IPT2V) generation aims to create high-fidelity videos with consistent human identity.<n>Current end-to-end frameworks suffer a critical spatial-temporal trade-off.<n>We propose a simple yet effective spatial-temporal decoupled framework that decomposes representations into spatial features for layouts and temporal features for motion dynamics.
  arXiv  Detail & Related papers  (2025-07-07T06:54:44Z)
• Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimization [43.81805412365791]
  We introduce direct preference optimization tailored for human-centric animation.<n>Second, the proposed temporal motion modulation resolves resolution mismatches.<n>Experiments demonstrate obvious improvements in lip-audio synchronization, expression vividness, body motion coherence over baseline methods.
  arXiv  Detail & Related papers  (2025-05-29T15:04:00Z)
• AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars [71.90109867684025]
  Whole-body audio-driven avatar pose and expression generation is a critical task for creating lifelike digital humans.<n>We propose AsynFusion, a novel framework that leverages diffusion transformers to achieve cohesive expression and gesture synthesis.<n>AsynFusion achieves state-of-the-art performance in generating real-time, synchronized whole-body animations.
  arXiv  Detail & Related papers  (2025-05-21T03:28:53Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation [48.94486508604052]
  We introduce a novel, customizable one-shot audio-driven talking face generation framework, named PortraitTalk.<n>Our proposed method utilizes a latent diffusion framework consisting of two main components: IdentityNet and AnimateNet.<n>Key innovation of PortraitTalk is the incorporation of text prompts through decoupled cross-attention mechanisms.
  arXiv  Detail & Related papers  (2024-12-10T18:51:31Z)
• Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation [29.87407471246318]
  This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations. Our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module. The proposed hierarchical audio-driven visual synthesis offers adaptive control over expression and pose diversity, enabling more effective personalization tailored to different identities.
  arXiv  Detail & Related papers  (2024-06-13T04:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.