Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedback

• URL: http://arxiv.org/abs/2510.12089v1
• Date: Tue, 14 Oct 2025 02:50:05 GMT
• Title: Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedback
• Authors: Xingpei Ma, Shenneng Huang, Jiaran Cai, Yuansheng Guan, Shen Zheng, Hanfeng Zhao, Qiang Zhang, Shunsi Zhang,
• Abstract summary: We propose a diffusion transformer (DiT)-based framework for generating talking videos of arbitrary length.<n>We also introduce a training-free method for multi-character audio-driven animation.<n> Experimental results demonstrate that our method outperforms existing state-of-the-art approaches.
• Score: 9.569613635896026
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in diffusion models have significantly improved audio-driven human video generation, surpassing traditional methods in both quality and controllability. However, existing approaches still face challenges in lip-sync accuracy, temporal coherence for long video generation, and multi-character animation. In this work, we propose a diffusion transformer (DiT)-based framework for generating lifelike talking videos of arbitrary length, and introduce a training-free method for multi-character audio-driven animation. First, we employ a LoRA-based training strategy combined with a position shift inference approach, which enables efficient long video generation while preserving the capabilities of the foundation model. Moreover, we combine partial parameter updates with reward feedback to enhance both lip synchronization and natural body motion. Finally, we propose a training-free approach, Mask Classifier-Free Guidance (Mask-CFG), for multi-character animation, which requires no specialized datasets or model modifications and supports audio-driven animation for three or more characters. Experimental results demonstrate that our method outperforms existing state-of-the-art approaches, achieving high-quality, temporally coherent, and multi-character audio-driven video generation in a simple, efficient, and cost-effective manner.

Related papers

• FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generation [23.08428760363473]
  FlowPortrait is a reinforcement-learning framework for audio-driven portrait animation.<n>It produces higher-quality talking-head videos, highlighting the effectiveness of reinforcement learning for portrait animation.
  arXiv  Detail & Related papers  (2026-02-25T22:08:15Z)
• EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer [64.69014756863331]
  We introduce EchoMotion, a framework designed to model the joint distribution of appearance and human motion.<n>We also propose MVS-RoPE, which offers unified 3D positional encoding for both video and motion tokens.<n>Our findings reveal that explicitly representing human motion is to appearance, significantly boosting the coherence and plausibility of human-centric video generation.
  arXiv  Detail & Related papers  (2025-12-21T17:08:14Z)
• PersonaLive! Expressive Portrait Image Animation for Live Streaming [53.63615310186964]
  PersonaLive is a novel diffusion-based framework towards streaming real-time portrait animation.<n>We first adopt hybrid implicit signals, namely implicit facial representations and 3D implicit keypoints, to achieve expressive image-level motion control.<n>Experiments demonstrate that PersonaLive achieves state-of-the-art performance with up to 7-22x speedup over prior diffusion-based portrait animation models.
  arXiv  Detail & Related papers  (2025-12-12T03:24:40Z)
• StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model [73.30619724574642]
  Speech-driven 3D facial animation aims to generate realistic and synchronized facial motions driven by speech inputs.<n>Recent methods have employed audio-conditioned diffusion models for 3D facial animation.<n>We propose a novel autoregressive diffusion model that processes audio in a streaming manner.
  arXiv  Detail & Related papers  (2025-11-18T07:55:16Z)
• Wan-S2V: Audio-Driven Cinematic Video Generation [53.59046547320306]
  Current state-of-the-art (SOTA) methods for audio-driven character animation demonstrate promising performance for scenarios involving speech and singing.<n>We propose an audio-driven model, which we refere to as Wan-S2V, built upon Wan.
  arXiv  Detail & Related papers  (2025-08-26T02:51:31Z)
• AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation [65.06374691172061]
  multimodal-to-speech task has gained increasing attention due to its wide range of applications, such as film production, dubbing, and virtual avatars.<n>Existing methods still suffer from limitations in speech intelligibility, audio-video synchronization, speech naturalness, and voice similarity to the reference speaker.<n>We propose AlignDiT, a multimodal Aligned Diffusion Transformer that generates accurate, synchronized, and natural-sounding speech from aligned multimodal inputs.
  arXiv  Detail & Related papers  (2025-04-29T10:56:24Z)
• EasyGenNet: An Efficient Framework for Audio-Driven Gesture Video Generation Based on Diffusion Model [22.286624353800377]
  We introduce a new audio-to-video pipeline to synthesize co-speech videos, using 2D human skeleton as the intermediate motion representation.<n>Our experiments show that our method outperforms existing GAN-based and diffusion-based methods.
  arXiv  Detail & Related papers  (2025-04-11T08:19:18Z)
• DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation [54.30327187663316]
  DiTCtrl is a training-free multi-prompt video generation method under MM-DiT architectures for the first time.<n>We analyze MM-DiT's attention mechanism, finding that the 3D full attention behaves similarly to that of the cross/self-attention blocks in the UNet-like diffusion models.<n>Based on our careful design, the video generated by DiTCtrl achieves smooth transitions and consistent object motion given multiple sequential prompts.
  arXiv  Detail & Related papers  (2024-12-24T18:51:19Z)
• TVG: A Training-free Transition Video Generation Method with Diffusion Models [12.037716102326993]
  Transition videos play a crucial role in media production, enhancing the flow and coherence of visual narratives. Recent advances in diffusion model-based video generation offer new possibilities for creating transitions but face challenges such as poor inter-frame relationship modeling and abrupt content changes. We propose a novel training-free Transition Video Generation (TVG) approach using video-level diffusion models that addresses these limitations without additional training.
  arXiv  Detail & Related papers  (2024-08-24T00:33:14Z)
• UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation [53.16986875759286]
  We present a UniAnimate framework to enable efficient and long-term human video generation. We map the reference image along with the posture guidance and noise video into a common feature space. We also propose a unified noise input that supports random noised input as well as first frame conditioned input.
  arXiv  Detail & Related papers  (2024-06-03T10:51:10Z)
• SAiD: Speech-driven Blendshape Facial Animation with Diffusion [6.4271091365094515]
  Speech-driven 3D facial animation is challenging due to the scarcity of large-scale visual-audio datasets. We propose a speech-driven 3D facial animation with a diffusion model (SAiD), a lightweight Transformer-based U-Net with a cross-modality alignment bias between audio and visual to enhance lip synchronization.
  arXiv  Detail & Related papers  (2023-12-25T04:40:32Z)
• Audio-Driven Dubbing for User Generated Contents via Style-Aware Semi-Parametric Synthesis [123.11530365315677]
  Existing automated dubbing methods are usually designed for Professionally Generated Content (PGC) production. In this paper, we investigate an audio-driven dubbing method that is more feasible for User Generated Content (UGC) production.
  arXiv  Detail & Related papers  (2023-08-31T15:41:40Z)
• Continuous-Time Video Generation via Learning Motion Dynamics with Neural ODE [26.13198266911874]
  We propose a novel video generation approach that learns separate distributions for motion and appearance. We employ a two-stage approach where the first stage converts a noise vector to a sequence of keypoints in arbitrary frame rates, and the second stage synthesizes videos based on the given keypoints sequence and the appearance noise vector.
  arXiv  Detail & Related papers  (2021-12-21T03:30:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.