AniFormer: Data-driven 3D Animation with Transformer
- URL: http://arxiv.org/abs/2110.10533v1
- Date: Wed, 20 Oct 2021 12:36:55 GMT
- Title: AniFormer: Data-driven 3D Animation with Transformer
- Authors: Haoyu Chen, Hao Tang, Nicu Sebe, Guoying Zhao
- Abstract summary: We present a novel task, i.e., animating a target 3D object through the motion of a raw driving sequence.
AniFormer generates animated 3D sequences by directly taking the raw driving sequences and arbitrary same-type target meshes as inputs.
Our AniFormer achieves high-fidelity, realistic, temporally coherent animated results and outperforms compared start-of-the-art methods on benchmarks of diverse categories.
- Score: 95.45760189583181
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a novel task, i.e., animating a target 3D object through the
motion of a raw driving sequence. In previous works, extra auxiliary
correlations between source and target meshes or intermedia factors are
inevitable to capture the motions in the driving sequences. Instead, we
introduce AniFormer, a novel Transformer-based architecture, that generates
animated 3D sequences by directly taking the raw driving sequences and
arbitrary same-type target meshes as inputs. Specifically, we customize the
Transformer architecture for 3D animation that generates mesh sequences by
integrating styles from target meshes and motions from the driving meshes.
Besides, instead of the conventional single regression head in the vanilla
Transformer, AniFormer generates multiple frames as outputs to preserve the
sequential consistency of the generated meshes. To achieve this, we carefully
design a pair of regression constraints, i.e., motion and appearance
constraints, that can provide strong regularization on the generated mesh
sequences. Our AniFormer achieves high-fidelity, realistic, temporally coherent
animated results and outperforms compared start-of-the-art methods on
benchmarks of diverse categories. Code is available:
https://github.com/mikecheninoulu/AniFormer.
Related papers
- AniDoc: Animation Creation Made Easier [54.97341104616779]
Our research focuses on reducing the labor costs in the production of 2D animation by harnessing the potential of increasingly powerful AI.
AniDoc emerges as a video line art colorization tool, which automatically converts sketch sequences into colored animations.
Our model exploits correspondence matching as an explicit guidance, yielding strong robustness to the variations between the reference character and each line art frame.
arXiv Detail & Related papers (2024-12-18T18:59:59Z) - Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks [25.39030226963548]
We introduce the first application of a pretrained transformer-based video generative model for portrait animation.
Our method is validated through experiments on benchmark and newly proposed wild datasets.
arXiv Detail & Related papers (2024-12-01T08:54:30Z) - Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics [67.97235923372035]
We present Puppet-Master, an interactive video generative model that can serve as a motion prior for part-level dynamics.
At test time, given a single image and a sparse set of motion trajectories, Puppet-Master can synthesize a video depicting realistic part-level motion faithful to the given drag interactions.
arXiv Detail & Related papers (2024-08-08T17:59:38Z) - Generating 3D House Wireframes with Semantics [11.408526398063712]
We present a new approach for generating 3D house with semantic enrichment using an autoregressive model.
By re-ordering wire sequences based on semantic meanings, we employ a seamless semantic sequence for learning on 3D wireframe structures.
arXiv Detail & Related papers (2024-07-17T02:33:34Z) - Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer [62.29951737214263]
Existing algorithms directly generate the full sequence which is expensive and prone to errors.
We propose KeyMotion, that generates plausible human motion sequences corresponding to input text.
We use a Variationalcoder (VAE) with Kullback-Leibler regularization to project the Autoencoder into a latent space.
For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the design latents and text condition.
arXiv Detail & Related papers (2024-05-24T11:12:37Z) - Hierarchical Generation of Human-Object Interactions with Diffusion
Probabilistic Models [71.64318025625833]
This paper presents a novel approach to generating the 3D motion of a human interacting with a target object.
Our framework first generates a set of milestones and then synthesizes the motion along them.
The experiments on the NSM, COUCH, and SAMP datasets show that our approach outperforms previous methods by a large margin in both quality and diversity.
arXiv Detail & Related papers (2023-10-03T17:50:23Z) - Robust Motion In-betweening [17.473287573543065]
We present a novel, robust transition generation technique that can serve as a new tool for 3D animators.
The system synthesizes high-quality motions that use temporally-sparsers as animation constraints.
We present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios.
arXiv Detail & Related papers (2021-02-09T16:52:45Z) - First Order Motion Model for Image Animation [90.712718329677]
Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video.
Our framework addresses this problem without using any annotation or prior information about the specific object to animate.
arXiv Detail & Related papers (2020-02-29T07:08:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.