AniFormer: Data-driven 3D Animation with Transformer

• URL: http://arxiv.org/abs/2110.10533v1
• Date: Wed, 20 Oct 2021 12:36:55 GMT
• Title: AniFormer: Data-driven 3D Animation with Transformer
• Authors: Haoyu Chen, Hao Tang, Nicu Sebe, Guoying Zhao
• Abstract summary: We present a novel task, i.e., animating a target 3D object through the motion of a raw driving sequence. AniFormer generates animated 3D sequences by directly taking the raw driving sequences and arbitrary same-type target meshes as inputs. Our AniFormer achieves high-fidelity, realistic, temporally coherent animated results and outperforms compared start-of-the-art methods on benchmarks of diverse categories.
• Score: 95.45760189583181
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel task, i.e., animating a target 3D object through the motion of a raw driving sequence. In previous works, extra auxiliary correlations between source and target meshes or intermedia factors are inevitable to capture the motions in the driving sequences. Instead, we introduce AniFormer, a novel Transformer-based architecture, that generates animated 3D sequences by directly taking the raw driving sequences and arbitrary same-type target meshes as inputs. Specifically, we customize the Transformer architecture for 3D animation that generates mesh sequences by integrating styles from target meshes and motions from the driving meshes. Besides, instead of the conventional single regression head in the vanilla Transformer, AniFormer generates multiple frames as outputs to preserve the sequential consistency of the generated meshes. To achieve this, we carefully design a pair of regression constraints, i.e., motion and appearance constraints, that can provide strong regularization on the generated mesh sequences. Our AniFormer achieves high-fidelity, realistic, temporally coherent animated results and outperforms compared start-of-the-art methods on benchmarks of diverse categories. Code is available: https://github.com/mikecheninoulu/AniFormer.

Related papers

• Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics [67.97235923372035]
  We present Puppet-Master, an interactive video generative model that can serve as a motion prior for part-level dynamics. At test time, given a single image and a sparse set of motion trajectories, Puppet-Master can synthesize a video depicting realistic part-level motion faithful to the given drag interactions.
  arXiv  Detail & Related papers  (2024-08-08T17:59:38Z)
• PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers [5.7181794813117754]
  PASTA is an autoregressive transformer architecture for generating high quality 3D shapes. Our model generates 3D shapes that are both more realistic and diverse than existing part-based and non part-based methods.
  arXiv  Detail & Related papers  (2024-07-18T16:52:45Z)
• Generating 3D House Wireframes with Semantics [11.408526398063712]
  We present a new approach for generating 3D house with semantic enrichment using an autoregressive model. By re-ordering wire sequences based on semantic meanings, we employ a seamless semantic sequence for learning on 3D wireframe structures.
  arXiv  Detail & Related papers  (2024-07-17T02:33:34Z)
• Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer [62.29951737214263]
  Existing algorithms directly generate the full sequence which is expensive and prone to errors. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text. We use a Variationalcoder (VAE) with Kullback-Leibler regularization to project the Autoencoder into a latent space. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the design latents and text condition.
  arXiv  Detail & Related papers  (2024-05-24T11:12:37Z)
• GenDeF: Learning Generative Deformation Field for Video Generation [89.49567113452396]
  We propose to render a video by warping one static image with a generative deformation field (GenDeF) Such a pipeline enjoys three appealing advantages.
  arXiv  Detail & Related papers  (2023-12-07T18:59:41Z)
• Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models [71.64318025625833]
  This paper presents a novel approach to generating the 3D motion of a human interacting with a target object. Our framework first generates a set of milestones and then synthesizes the motion along them. The experiments on the NSM, COUCH, and SAMP datasets show that our approach outperforms previous methods by a large margin in both quality and diversity.
  arXiv  Detail & Related papers  (2023-10-03T17:50:23Z)
• Robust Motion In-betweening [17.473287573543065]
  We present a novel, robust transition generation technique that can serve as a new tool for 3D animators. The system synthesizes high-quality motions that use temporally-sparsers as animation constraints. We present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios.
  arXiv  Detail & Related papers  (2021-02-09T16:52:45Z)
• First Order Motion Model for Image Animation [90.712718329677]
  Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate.
  arXiv  Detail & Related papers  (2020-02-29T07:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.