Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation

• URL: http://arxiv.org/abs/2512.13495v1
• Date: Mon, 15 Dec 2025 16:25:56 GMT
• Title: Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation
• Abstract summary: Soul is a framework for high-fidelity long-term digital human animation.<n>It generates semantically coherent videos from a single-frame portrait image, text prompts, and audio.<n>It achieves precise lip synchronization, vivid facial expressions, and robust identity preservation.
• Score: 70.86886231976439
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a multimodal-driven framework for high-fidelity long-term digital human animation termed $\textbf{Soul}$, which generates semantically coherent videos from a single-frame portrait image, text prompts, and audio, achieving precise lip synchronization, vivid facial expressions, and robust identity preservation. We construct Soul-1M, containing 1 million finely annotated samples with a precise automated annotation pipeline (covering portrait, upper-body, full-body, and multi-person scenes) to mitigate data scarcity, and we carefully curate Soul-Bench for comprehensive and fair evaluation of audio-/text-guided animation methods. The model is built on the Wan2.2-5B backbone, integrating audio-injection layers and multiple training strategies together with threshold-aware codebook replacement to ensure long-term generation consistency. Meanwhile, step/CFG distillation and a lightweight VAE are used to optimize inference efficiency, achieving an 11.4$\times$ speedup with negligible quality loss. Extensive experiments show that Soul significantly outperforms current leading open-source and commercial models on video quality, video-text alignment, identity preservation, and lip-synchronization accuracy, demonstrating broad applicability in real-world scenarios such as virtual anchors and film production. Project page at https://zhangzjn.github.io/projects/Soul/

Related papers

• Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds [60.18437892555415]
  We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants.<n>The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio.<n>The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway.
  arXiv  Detail & Related papers  (2026-08-24T15:31:12Z)
• EverAnimate: Minute-Scale Human Animation via Latent Flow Restoration [74.49189147054337]
  We propose EverAnimate, an efficient post-training method for long-horizon animated video generation.<n>EverAnimate restores drifted flow trajectories by anchoring generation to a persistent latent context memory.<n>With only lightweight LoRA tuning, EverAnimate outperforms state-of-the-art long-animation methods in both short- and long-horizon settings.
  arXiv  Detail & Related papers  (2026-05-14T16:36:34Z)
• PEARL: Personalized Streaming Video Understanding Model [50.273809779498464]
  multimodal personalization methods are largely limited to static images or offline videos.<n>PEARL-Bench is the first comprehensive benchmark designed specifically to evaluate this challenging setting.<n>PEARL achieves state-of-the-art performance across 8 offline and online models.
  arXiv  Detail & Related papers  (2026-03-20T18:47:54Z)
• FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes [56.534404169212785]
  FunCineForge comprises an end-to-end production pipeline for large-scale dubbing datasets and an MLLM-based dubbing model designed for diverse cinematic scenes.<n>We construct the first Chinese television dubbing dataset with rich annotations, and demonstrate the high quality of these data.<n>Experiments across monologue, narration, dialogue, and multi-speaker scenes show that our dubbing model consistently outperforms SOTA methods in audio quality, lip sync, timbre transfer, and instruction following.
  arXiv  Detail & Related papers  (2026-01-21T08:57:00Z)
• From Inpainting to Editing: A Self-Bootstrapping Framework for Context-Rich Visual Dubbing [24.998261989251976]
  We propose a self-bootstrapping framework that reframes visual dubbing from an ill-posed inpainting task into a well-conditioned video-to-video editing problem.<n>Our approach employs a Diffusion Transformer, first as a data generator, to synthesize ideal training data.<n>A DiDubT-based audio-driven editor is then trained on these pairs end-to-end, leveraging the complete input video frames to focus solely on precise, audio-driven lip modifications.
  arXiv  Detail & Related papers  (2025-12-31T18:58:30Z)
• Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedback [9.569613635896026]
  We propose a diffusion transformer (DiT)-based framework for generating talking videos of arbitrary length.<n>We also introduce a training-free method for multi-character audio-driven animation.<n> Experimental results demonstrate that our method outperforms existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-10-14T02:50:05Z)
• StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbing [63.72095377128904]
  The visual dubbing task aims to generate mouth movements synchronized with the driving audio.<n>Audio-only driving paradigms inadequately capture speaker-specific lip habits.<n>Blind-inpainting approaches produce visual artifacts when handling obstructions.
  arXiv  Detail & Related papers  (2025-09-26T05:23:31Z)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing [66.48064661467781]
  We introduce sparse-frame video dubbing, a novel paradigm that strategically preserves references to maintain identity, iconic gestures, and camera trajectories.<n>We propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing.<n> Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-19T17:55:23Z)
• JWB-DH-V1: Benchmark for Joint Whole-Body Talking Avatar and Speech Generation Version 1 [6.4645943969421875]
  This paper introduces the Joint Whole-Body Talking Avatar and Speech Generation Version I(JWB-DH-V1)<n>It comprises a large-scale multi-modal dataset with 10,000 unique identities across 2 million video samples, and an evaluation protocol for assessing joint audio-video generation of whole-body animatable avatars.<n>Our evaluation of SOTA models reveals consistent performance disparities between face/hand-centric and whole-body performance.
  arXiv  Detail & Related papers  (2025-07-28T16:47:44Z)
• AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation [52.655400705690155]
  AnimeShooter is a reference-guided multi-shot animation dataset.<n>Story-level annotations provide an overview of the narrative, including the storyline, key scenes, and main character profiles with reference images.<n>Shot-level annotations decompose the story into consecutive shots, each annotated with scene, characters, and both narrative and descriptive visual captions.<n>A separate subset, AnimeShooter-audio, offers synchronized audio tracks for each shot, along with audio descriptions and sound sources.
  arXiv  Detail & Related papers  (2025-06-03T17:55:18Z)
• SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers [25.36460340267922]
  We present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos.<n>Our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs.
  arXiv  Detail & Related papers  (2025-06-01T04:27:13Z)
• Efficient Long-duration Talking Video Synthesis with Linear Diffusion Transformer under Multimodal Guidance [39.94595889521696]
  LetsTalk is a diffusion transformer framework equipped with multimodal guidance and a novel memory bank mechanism.<n>In particular, LetsTalk introduces a noise-regularized memory bank to alleviate error accumulation and sampling artifacts during extended video generation.<n>We show that LetsTalk establishes new state-of-the-art in generation quality, producing temporally coherent and realistic talking videos.
  arXiv  Detail & Related papers  (2024-11-24T04:46:00Z)
• UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation [53.16986875759286]
  We present a UniAnimate framework to enable efficient and long-term human video generation. We map the reference image along with the posture guidance and noise video into a common feature space. We also propose a unified noise input that supports random noised input as well as first frame conditioned input.
  arXiv  Detail & Related papers  (2024-06-03T10:51:10Z)
• Do You Guys Want to Dance: Zero-Shot Compositional Human Dance Generation with Multiple Persons [73.21855272778616]
  We introduce a new task, dataset, and evaluation protocol of compositional human dance generation (cHDG) We propose a novel zero-shot framework, dubbed MultiDance-Zero, that can synthesize videos consistent with arbitrary multiple persons and background while precisely following the driving poses.
  arXiv  Detail & Related papers  (2024-01-24T10:44:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.