DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds

• URL: http://arxiv.org/abs/2505.24733v1
• Date: Fri, 30 May 2025 15:54:34 GMT
• Title: DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds
• Authors: Jiaxu Zhang, Xianfang Zeng, Xin Chen, Wei Zuo, Gang Yu, Guosheng Lin, Zhigang Tu,
• Abstract summary: DreamDance is an animation framework capable of producing stable, consistent character and scene motion conditioned on precise camera trajectories.<n>We train a pose-aware video inpainting model that injects the dynamic character into the scene video while enhancing background quality.
• Score: 64.53681498600065
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents DreamDance, a novel character art animation framework capable of producing stable, consistent character and scene motion conditioned on precise camera trajectories. To achieve this, we re-formulate the animation task as two inpainting-based steps: Camera-aware Scene Inpainting and Pose-aware Video Inpainting. The first step leverages a pre-trained image inpainting model to generate multi-view scene images from the reference art and optimizes a stable large-scale Gaussian field, which enables coarse background video rendering with camera trajectories. However, the rendered video is rough and only conveys scene motion. To resolve this, the second step trains a pose-aware video inpainting model that injects the dynamic character into the scene video while enhancing background quality. Specifically, this model is a DiT-based video generation model with a gating strategy that adaptively integrates the character's appearance and pose information into the base background video. Through extensive experiments, we demonstrate the effectiveness and generalizability of DreamDance, producing high-quality and consistent character animations with remarkable camera dynamics.

Related papers

• FairyGen: Storied Cartoon Video from a Single Child-Drawn Character [15.701180508477679]
  We propose FairyGen, an automatic system for generating story-driven cartoon videos from a single child's drawing.<n>Unlike previous storytelling methods, FairyGen explicitly disentangles character modeling from stylized background generation.<n>Our system produces animations that are stylistically faithful, narratively structured natural motion.
  arXiv  Detail & Related papers  (2025-06-26T13:58:16Z)
• Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes [49.26872036160368]
  We propose a method for animating parts of high-quality 3D scenes in a Gaussian Splatting representation.<n>We find that, in contrast to prior work, this enables realistic animations of complex, pre-existing 3D scenes.
  arXiv  Detail & Related papers  (2024-11-28T16:01:58Z)
• Replace Anyone in Videos [82.37852750357331]
  We present the ReplaceAnyone framework, which focuses on localized human replacement and insertion featuring intricate backgrounds.<n>We formulate this task as an image-conditioned video inpainting paradigm with pose guidance, utilizing a unified end-to-end video diffusion architecture.<n>The proposed ReplaceAnyone can be seamlessly applied not only to traditional 3D-UNet base models but also to DiT-based video models such as Wan2.1.
  arXiv  Detail & Related papers  (2024-09-30T03:27:33Z)
• AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment [64.02822911038848]
  We present AnimateZoo, a zero-shot diffusion-based video generator to produce animal animations. Key technique used in our AnimateZoo is subject alignment, which includes two steps. Our model is capable of generating videos characterized by accurate movements, consistent appearance, and high-fidelity frames.
  arXiv  Detail & Related papers  (2024-04-07T12:57:41Z)
• AnimateZero: Video Diffusion Models are Zero-Shot Image Animators [63.938509879469024]
  We propose AnimateZero to unveil the pre-trained text-to-video diffusion model, i.e., AnimateDiff. For appearance control, we borrow intermediate latents and their features from the text-to-image (T2I) generation. For temporal control, we replace the global temporal attention of the original T2V model with our proposed positional-corrected window attention.
  arXiv  Detail & Related papers  (2023-12-06T13:39:35Z)
• Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation [27.700371215886683]
  diffusion models have become the mainstream in visual generation research, owing to their robust generative capabilities. In this paper, we propose a novel framework tailored for character animation. By expanding the training data, our approach can animate arbitrary characters, yielding superior results in character animation compared to other image-to-video methods.
  arXiv  Detail & Related papers  (2023-11-28T12:27:15Z)
• MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model [74.84435399451573]
  This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. We introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity.
  arXiv  Detail & Related papers  (2023-11-27T18:32:31Z)
• AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning [92.33690050667475]
  AnimateDiff is a framework for animating personalized T2I models without requiring model-specific tuning. We propose MotionLoRA, a lightweight fine-tuning technique for AnimateDiff that enables a pre-trained motion module to adapt to new motion patterns. Results show that our approaches help these models generate temporally smooth animation clips while preserving the visual quality and motion diversity.
  arXiv  Detail & Related papers  (2023-07-10T17:34:16Z)
• Going beyond Free Viewpoint: Creating Animatable Volumetric Video of Human Performances [7.7824496657259665]
  We present an end-to-end pipeline for the creation of high-quality animatable volumetric video content of human performances. Semantic enrichment and geometric animation ability are achieved by establishing temporal consistency in the 3D data. For pose editing, we exploit the captured data as much as possible and kinematically deform the captured frames to fit a desired pose.
  arXiv  Detail & Related papers  (2020-09-02T09:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.