Motion Avatar: Generate Human and Animal Avatars with Arbitrary Motion

• URL: http://arxiv.org/abs/2405.11286v2
• Date: Fri, 30 Aug 2024 19:17:41 GMT
• Title: Motion Avatar: Generate Human and Animal Avatars with Arbitrary Motion
• Authors: Zeyu Zhang, Yiran Wang, Biao Wu, Shuo Chen, Zhiyuan Zhang, Shiya Huang, Wenbo Zhang, Meng Fang, Ling Chen, Yang Zhao,
• Abstract summary: We propose a novel agent-based approach named Motion Avatar, which allows for the automatic generation of high-quality customizable human and animal avatars. Secondly, we introduced a LLM planner that coordinates both motion and avatar generation, which transforms a discriminative planning into a customizable Q&A fashion. Finally, we presented an animal motion dataset named Zoo-300K, comprising approximately 300,000 text-motion pairs across 65 animal categories.
• Score: 39.456643736018435
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In recent years, there has been significant interest in creating 3D avatars and motions, driven by their diverse applications in areas like film-making, video games, AR/VR, and human-robot interaction. However, current efforts primarily concentrate on either generating the 3D avatar mesh alone or producing motion sequences, with integrating these two aspects proving to be a persistent challenge. Additionally, while avatar and motion generation predominantly target humans, extending these techniques to animals remains a significant challenge due to inadequate training data and methods. To bridge these gaps, our paper presents three key contributions. Firstly, we proposed a novel agent-based approach named Motion Avatar, which allows for the automatic generation of high-quality customizable human and animal avatars with motions through text queries. The method significantly advanced the progress in dynamic 3D character generation. Secondly, we introduced a LLM planner that coordinates both motion and avatar generation, which transforms a discriminative planning into a customizable Q&A fashion. Lastly, we presented an animal motion dataset named Zoo-300K, comprising approximately 300,000 text-motion pairs across 65 animal categories and its building pipeline ZooGen, which serves as a valuable resource for the community. See project website https://steve-zeyu-zhang.github.io/MotionAvatar/

Related papers

• AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation [60.5897687447003]
  AvatarGO is a novel framework designed to generate realistic 4D HOI scenes from textual inputs. Our framework not only generates coherent compositional motions, but also exhibits greater robustness in handling issues. As the first attempt to synthesize 4D avatars with object interactions, we hope AvatarGO could open new doors for human-centric 4D content creation.
  arXiv  Detail & Related papers  (2024-10-09T17:58:56Z)
• AMG: Avatar Motion Guided Video Generation [5.82136706118236]
  We propose AMG, a method that combines the 2D photorealism and 3D controllability by conditioning video diffusion models on controlled rendering of 3D avatars. AMG is the first method that enables multi-person diffusion video generation with precise control over camera positions, human motions, and background style.
  arXiv  Detail & Related papers  (2024-09-02T23:59:01Z)
• HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation [64.37874983401221]
  We present HumanVid, the first large-scale high-quality dataset tailored for human image animation. For the real-world data, we compile a vast collection of real-world videos from the internet. For the synthetic data, we collected 10K 3D avatar assets and leveraged existing assets of body shapes, skin textures and clothings.
  arXiv  Detail & Related papers  (2024-07-24T17:15:58Z)
• AniArtAvatar: Animatable 3D Art Avatar from a Single Image [0.0]
  We present a novel approach for generating animatable 3D-aware art avatars from a single image. We use a view-conditioned 2D diffusion model to synthesize multi-view images from a single art portrait with a neutral expression. For avatar animation, we extract control points, transfer the motion with these points, and deform the implicit canonical space.
  arXiv  Detail & Related papers  (2024-03-26T12:08:04Z)
• OmniMotionGPT: Animal Motion Generation with Limited Data [70.35662376853163]
  We introduce AnimalML3D, the first text-animal motion dataset with 1240 animation sequences spanning 36 different animal identities. We are able to generate animal motions with high diversity and fidelity, quantitatively and qualitatively outperforming the results of training human motion generation baselines on animal data.
  arXiv  Detail & Related papers  (2023-11-30T07:14:00Z)
• MagicAvatar: Multimodal Avatar Generation and Animation [70.55750617502696]
  MagicAvatar is a framework for multimodal video generation and animation of human avatars. It disentangles avatar video generation into two stages: multimodal-to-motion and motion-to-video generation. We demonstrate the flexibility of MagicAvatar through various applications, including text-guided and video-guided avatar generation.
  arXiv  Detail & Related papers  (2023-08-28T17:56:18Z)
• Physics-based Motion Retargeting from Sparse Inputs [73.94570049637717]
  Commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available.
  arXiv  Detail & Related papers  (2023-07-04T21:57:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.