Versatile Multimodal Controls for Expressive Talking Human Animation

• URL: http://arxiv.org/abs/2503.08714v3
• Date: Wed, 16 Apr 2025 02:43:12 GMT
• Title: Versatile Multimodal Controls for Expressive Talking Human Animation
• Authors: Zheng Qin, Ruobing Zheng, Yabing Wang, Tianqi Li, Zixin Zhu, Sanping Zhou, Ming Yang, Le Wang,
• Abstract summary: VersaAnimator is a versatile framework that synthesizes expressive talking human videos from arbitrary portrait images.<n>We introduce a token2pose translator to smoothly map 3D motion tokens to 2D pose sequences.
• Score: 26.61771541877306
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In filmmaking, directors typically allow actors to perform freely based on the script before providing specific guidance on how to present key actions. AI-generated content faces similar requirements, where users not only need automatic generation of lip synchronization and basic gestures from audio input but also desire semantically accurate and expressive body movement that can be ``directly guided'' through text descriptions. Therefore, we present VersaAnimator, a versatile framework that synthesizes expressive talking human videos from arbitrary portrait images. Specifically, we design a motion generator that produces basic rhythmic movements from audio input and supports text-prompt control for specific actions. The generated whole-body 3D motion tokens can animate portraits of various scales, producing talking heads, half-body gestures and even leg movements for whole-body images. Besides, we introduce a multi-modal controlled video diffusion that generates photorealistic videos, where speech signals govern lip synchronization, facial expressions, and head motions while body movements are guided by the 2D poses. Furthermore, we introduce a token2pose translator to smoothly map 3D motion tokens to 2D pose sequences. This design mitigates the stiffness resulting from direct 3D to 2D conversion and enhances the details of the generated body movements. Extensive experiments shows that VersaAnimator synthesizes lip-synced and identity-preserving videos while generating expressive and semantically meaningful whole-body motions.

Related papers

• FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis [12.987186425491242]
  We propose a novel framework to generate high-fidelity, coherent talking portraits with controllable motion dynamics. In the first stage, we employ a clip-level training scheme to establish coherent global motion. In the second stage, we refine lip movements at the frame level using a lip-tracing mask, ensuring precise synchronization with audio signals.
  arXiv  Detail & Related papers  (2025-04-07T08:56:01Z)
• EMO2: End-Effector Guided Audio-Driven Avatar Video Generation [17.816939983301474]
  We propose a novel audio-driven talking head method capable of simultaneously generating highly expressive facial expressions and hand gestures.<n>In the first stage, we generate hand poses directly from audio input, leveraging the strong correlation between audio signals and hand movements.<n>In the second stage, we employ a diffusion model to synthesize video frames, incorporating the hand poses generated in the first stage to produce realistic facial expressions and body movements.
  arXiv  Detail & Related papers  (2025-01-18T07:51:29Z)
• Move-in-2D: 2D-Conditioned Human Motion Generation [54.067588636155115]
  We propose Move-in-2D, a novel approach to generate human motion sequences conditioned on a scene image.<n>Our approach accepts both a scene image and text prompt as inputs, producing a motion sequence tailored to the scene.
  arXiv  Detail & Related papers  (2024-12-17T18:58:07Z)
• LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement [8.973545189395953]
  This study focuses on the creation of visually compelling, time-synchronized animations through diffusion-based techniques. We process audio features separately and derive the corresponding control gates, which implicitly govern the movements in the mouth, eyes, and head, irrespective of the portrait's origin. The significant improvements in the fidelity of animated portraits, the accuracy of lip-syncing, and the appropriate motion variations achieved by our method render it a versatile tool for animating any portrait in any language.
  arXiv  Detail & Related papers  (2024-07-26T08:30:06Z)
• Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation [71.08922726494842]
  We introduce the problem of timeline control for text-driven motion synthesis. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. We propose a new test-time denoising method to generate composite animations from a multi-track timeline.
  arXiv  Detail & Related papers  (2024-01-16T18:39:15Z)
• 3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing [22.30870274645442]
  We present 3DiFACE, a novel method for personalized speech-driven 3D facial animation and editing. Our method outperforms existing state-of-the-art techniques and yields speech-driven animations with greater fidelity and diversity.
  arXiv  Detail & Related papers  (2023-12-01T19:01:05Z)
• DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion [68.85904927374165]
  We propose DF-3DFace, a diffusion-driven speech-to-3D face mesh synthesis. It captures the complex one-to-many relationships between speech and 3D face based on diffusion. It simultaneously achieves more realistic facial animation than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-23T04:14:55Z)
• Learning Audio-Driven Viseme Dynamics for 3D Face Animation [17.626644507523963]
  We present a novel audio-driven facial animation approach that can generate realistic lip-synchronized 3D animations from the input audio. Our approach learns viseme dynamics from speech videos, produces animator-friendly viseme curves, and supports multilingual speech inputs.
  arXiv  Detail & Related papers  (2023-01-15T09:55:46Z)
• Generating Holistic 3D Human Motion from Speech [97.11392166257791]
  We build a high-quality dataset of 3D holistic body meshes with synchronous speech. We then define a novel speech-to-motion generation framework in which the face, body, and hands are modeled separately.
  arXiv  Detail & Related papers  (2022-12-08T17:25:19Z)
• Audio-Driven Co-Speech Gesture Video Generation [92.15661971086746]
  We define and study this challenging problem of audio-driven co-speech gesture video generation. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. We propose a novel framework, Audio-driveN Gesture vIdeo gEneration (ANGIE), to effectively capture the reusable co-speech gesture patterns.
  arXiv  Detail & Related papers  (2022-12-05T15:28:22Z)
• StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation [47.06075725469252]
  StyleTalker is an audio-driven talking head generation model. It can synthesize a video of a talking person from a single reference image. Our model is able to synthesize talking head videos with impressive perceptual quality.
  arXiv  Detail & Related papers  (2022-08-23T12:49:01Z)
• MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement [142.9900055577252]
  We propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. Our approach ensures highly accurate lip motion, while also plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion.
  arXiv  Detail & Related papers  (2021-04-16T17:05:40Z)
• Synthesis of Compositional Animations from Textual Descriptions [54.85920052559239]
  "How unstructured and complex can we make a sentence and still generate plausible movements from it?" "How can we animate 3D-characters from a movie script or move robots by simply telling them what we would like them to do?"
  arXiv  Detail & Related papers  (2021-03-26T18:23:29Z)
• Learning Speech-driven 3D Conversational Gestures from Video [106.15628979352738]
  We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures. Our algorithm uses a CNN architecture that leverages the inherent correlation between facial expression and hand gestures. We also contribute a new way to create a large corpus of more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people.
  arXiv  Detail & Related papers  (2021-02-13T01:05:39Z)
• Audio- and Gaze-driven Facial Animation of Codec Avatars [149.0094713268313]
  We describe the first approach to animate Codec Avatars in real-time using audio and/or eye tracking. Our goal is to display expressive conversations between individuals that exhibit important social signals.
  arXiv  Detail & Related papers  (2020-08-11T22:28:48Z)
• Speech2Video Synthesis with 3D Skeleton Regularization and Expressive Body Poses [36.00309828380724]
  We propose a novel approach to convert given speech audio to a photo-realistic speaking video of a specific person. We achieve this by first generating 3D skeleton movements from the audio sequence using a recurrent neural network (RNN) To make the skeleton movement realistic and expressive, we embed the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic gestures into the generation process.
  arXiv  Detail & Related papers  (2020-07-17T19:30:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.