AniFormer: Data-driven 3D Animation with Transformer

• URL: http://arxiv.org/abs/2110.10533v1
• Date: Wed, 20 Oct 2021 12:36:55 GMT
• Title: AniFormer: Data-driven 3D Animation with Transformer
• Abstract summary: We present a novel task, i.e., animating a target 3D object through the motion of a raw driving sequence. AniFormer generates animated 3D sequences by directly taking the raw driving sequences and arbitrary same-type target meshes as inputs. Our AniFormer achieves high-fidelity, realistic, temporally coherent animated results and outperforms compared start-of-the-art methods on benchmarks of diverse categories.
• Score: 95.45760189583181
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel task, i.e., animating a target 3D object through the motion of a raw driving sequence. In previous works, extra auxiliary correlations between source and target meshes or intermedia factors are inevitable to capture the motions in the driving sequences. Instead, we introduce AniFormer, a novel Transformer-based architecture, that generates animated 3D sequences by directly taking the raw driving sequences and arbitrary same-type target meshes as inputs. Specifically, we customize the Transformer architecture for 3D animation that generates mesh sequences by integrating styles from target meshes and motions from the driving meshes. Besides, instead of the conventional single regression head in the vanilla Transformer, AniFormer generates multiple frames as outputs to preserve the sequential consistency of the generated meshes. To achieve this, we carefully design a pair of regression constraints, i.e., motion and appearance constraints, that can provide strong regularization on the generated mesh sequences. Our AniFormer achieves high-fidelity, realistic, temporally coherent animated results and outperforms compared start-of-the-art methods on benchmarks of diverse categories. Code is available: https://github.com/mikecheninoulu/AniFormer.

Related papers

• 3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement [51.22216867938034]
  We present a 3D motion animation framework that controls both human motion trajectories within a reconstructed 3D environment for video generation.<n>We design a viewpoint-adaptive latent fusion mechanism to inject point-cloud geometric priors through scene-visibility masking into the generative process.<n>Experiments on two standard human image animation benchmark datasets demonstrate remarkable improvements of our method over the state of the arts in related video generation metics.
  arXiv  Detail & Related papers  (2026-06-29T16:22:54Z)
• Generative Animations: A Multi-Model Pipeline for Prompt-Driven Motion Synthesis [0.688204255655161]
  We introduce Generative Animations, a system that transforms natural language prompts into production-ready animations.<n>By chaining Large Language Models (LLMs) for semantic parsing with the Segment Anything Model (SAM) for visual grounding, our pipeline automatically generates motion paths.<n>We demonstrate the system through three use cases: contour-following trajectories, orbital animations with z-order awareness, and perspective-aligned motion on transformed objects.
  arXiv  Detail & Related papers  (2026-05-26T15:58:30Z)
• R-DMesh: Video-Guided 3D Animation via Rectified Dynamic Mesh Flow [50.93811008612562]
  Video-guided 3D animation holds immense potential for content creation, offering intuitive and precise control over dynamic assets.<n>In real-world scenarios, the initial pose of a user-provided video rarely aligns with the starting frame of a reference video.<n>We propose R-DMesh, a framework designed to generate high-fidelity 4D meshes that align with video context.
  arXiv  Detail & Related papers  (2026-05-13T17:58:13Z)
• LottieGPT: Tokenizing Vector Animation for Autoregressive Generation [63.27046904946992]
  Vector animations offer resolution-independence, compactness, semantic structure, and parametric editable motion representations.<n>Current generative models operate exclusively in space and thus cannot synthesize them.<n>We finetune Qwen-VL to create LottieGPT, a native multimodal model capable of generating coherent, editable vector animations.
  arXiv  Detail & Related papers  (2026-04-13T17:55:40Z)
• DragMesh: Interactive 3D Generation Made Easy [12.832539752284466]
  DragMesh is a robust framework for real-time interactive 3D articulation.<n>Our core contribution is a novel decoupled kinematic reasoning and motion generation framework.
  arXiv  Detail & Related papers  (2025-12-06T13:10:44Z)
• Gaussian See, Gaussian Do: Semantic 3D Motion Transfer from Multiview Video [15.994811723477973]
  We present a novel approach for semantic 3D motion transfer from multiview video.<n>We extract motion embeddings from source videos via condition inversion, apply them to rendered frames, and use resulting videos to supervise dynamic 3D Gaussian Splatting reconstruction.<n>We establish the first benchmark for semantic 3D motion transfer and demonstrate superior motion fidelity and structural consistency compared to adapted baselines.
  arXiv  Detail & Related papers  (2025-11-18T19:02:50Z)
• Scaling Sequence-to-Sequence Generative Neural Rendering [37.23230422802279]
  Kaleido is a family of generative models designed for photorealistic, unified object- and scene-level neural rendering.<n>We introduce key architectural innovations that enable our model to:.<n>perform generative view synthesis without explicit 3D representations;.<n>generate any number of 6-DoF target views conditioned on any number of reference views;.<n> seamlessly unify 3D and video modelling within a single decoder-only rectified flow transformer.
  arXiv  Detail & Related papers  (2025-10-05T15:03:31Z)
• Puppeteer: Rig and Animate Your 3D Models [105.11046762553121]
  Puppeteer is a comprehensive framework that addresses both automatic rigging and animation for diverse 3D objects.<n>Our system first predicts plausible skeletal structures via an auto-regressive transformer.<n>It then infers skinning weights via an attention-based architecture.
  arXiv  Detail & Related papers  (2025-08-14T17:59:31Z)
• STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer [72.88105562624838]
  We present STream3R, a novel approach to 3D reconstruction that reformulates pointmap prediction as a decoder-only Transformer problem.<n>By learning geometric priors from large-scale 3D datasets, STream3R generalizes well to diverse and challenging scenarios.<n>Our results underscore the potential of causal Transformer models for online 3D perception, paving the way for real-time 3D understanding in streaming environments.
  arXiv  Detail & Related papers  (2025-08-14T17:58:05Z)
• Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video [19.830248504692563]
  DriveAnyMesh is a method for driving mesh guided by monocular video.<n>We present a 4D diffusion model that denoises sequences of latent sets.<n>Lashed sets leverage variational autoencoder, simultaneously capturing 3D shape and motion information.
  arXiv  Detail & Related papers  (2025-06-09T07:08:58Z)
• In-2-4D: Inbetweening from Two Single-View Images to 4D Generation [54.62824686338408]
  We propose a new problem, In-between2-4D, for generative 4D (i.e., 3D + motion) in Splating from a minimalistic input setting. Given two images representing the start and end states of an object in motion, our goal is to generate and reconstruct the motion in 4D.
  arXiv  Detail & Related papers  (2025-04-11T09:01:09Z)
• AniDoc: Animation Creation Made Easier [54.97341104616779]
  Our research focuses on reducing the labor costs in the production of 2D animation by harnessing the potential of increasingly powerful AI. AniDoc emerges as a video line art colorization tool, which automatically converts sketch sequences into colored animations. Our model exploits correspondence matching as an explicit guidance, yielding strong robustness to the variations between the reference character and each line art frame.
  arXiv  Detail & Related papers  (2024-12-18T18:59:59Z)
• Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer [25.39030226963548]
  We introduce the first application of a pretrained transformer-based video generative model for portrait animation. Our method is validated through experiments on benchmark and newly proposed wild datasets.
  arXiv  Detail & Related papers  (2024-12-01T08:54:30Z)
• Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics [67.97235923372035]
  We present Puppet-Master, an interactive video generative model that can serve as a motion prior for part-level dynamics. At test time, given a single image and a sparse set of motion trajectories, Puppet-Master can synthesize a video depicting realistic part-level motion faithful to the given drag interactions.
  arXiv  Detail & Related papers  (2024-08-08T17:59:38Z)
• PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers [5.7181794813117754]
  PASTA is an autoregressive transformer architecture for generating high quality 3D shapes. Our model generates 3D shapes that are both more realistic and diverse than existing part-based and non part-based methods.
  arXiv  Detail & Related papers  (2024-07-18T16:52:45Z)
• Generating 3D House Wireframes with Semantics [11.408526398063712]
  We present a new approach for generating 3D house with semantic enrichment using an autoregressive model. By re-ordering wire sequences based on semantic meanings, we employ a seamless semantic sequence for learning on 3D wireframe structures.
  arXiv  Detail & Related papers  (2024-07-17T02:33:34Z)
• Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer [62.29951737214263]
  Existing algorithms directly generate the full sequence which is expensive and prone to errors. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text. We use a Variationalcoder (VAE) with Kullback-Leibler regularization to project the Autoencoder into a latent space. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the design latents and text condition.
  arXiv  Detail & Related papers  (2024-05-24T11:12:37Z)
• GenDeF: Learning Generative Deformation Field for Video Generation [89.49567113452396]
  We propose to render a video by warping one static image with a generative deformation field (GenDeF) Such a pipeline enjoys three appealing advantages.
  arXiv  Detail & Related papers  (2023-12-07T18:59:41Z)
• Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models [71.64318025625833]
  This paper presents a novel approach to generating the 3D motion of a human interacting with a target object. Our framework first generates a set of milestones and then synthesizes the motion along them. The experiments on the NSM, COUCH, and SAMP datasets show that our approach outperforms previous methods by a large margin in both quality and diversity.
  arXiv  Detail & Related papers  (2023-10-03T17:50:23Z)
• Robust Motion In-betweening [17.473287573543065]
  We present a novel, robust transition generation technique that can serve as a new tool for 3D animators. The system synthesizes high-quality motions that use temporally-sparsers as animation constraints. We present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios.
  arXiv  Detail & Related papers  (2021-02-09T16:52:45Z)
• First Order Motion Model for Image Animation [90.712718329677]
  Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate.
  arXiv  Detail & Related papers  (2020-02-29T07:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.