MikuDance: Animating Character Art with Mixed Motion Dynamics

• URL: http://arxiv.org/abs/2411.08656v2
• Date: Thu, 14 Nov 2024 14:11:06 GMT
• Title: MikuDance: Animating Character Art with Mixed Motion Dynamics
• Authors: Jiaxu Zhang, Xianfang Zeng, Xin Chen, Wei Zuo, Gang Yu, Zhigang Tu,
• Abstract summary: We propose MikuDance, a diffusion-based pipeline incorporating mixed motion dynamics to animate character art. Specifically, a Scene Motion Tracking strategy is presented to explicitly model the dynamic camera in pixel-wise space, enabling unified character-scene motion modeling. A Motion-Adaptive Normalization module is incorporated to effectively inject global scene motion, paving the way for comprehensive character art animation.
• Score: 28.189884806755153
• License:
• Abstract: We propose MikuDance, a diffusion-based pipeline incorporating mixed motion dynamics to animate stylized character art. MikuDance consists of two key techniques: Mixed Motion Modeling and Mixed-Control Diffusion, to address the challenges of high-dynamic motion and reference-guidance misalignment in character art animation. Specifically, a Scene Motion Tracking strategy is presented to explicitly model the dynamic camera in pixel-wise space, enabling unified character-scene motion modeling. Building on this, the Mixed-Control Diffusion implicitly aligns the scale and body shape of diverse characters with motion guidance, allowing flexible control of local character motion. Subsequently, a Motion-Adaptive Normalization module is incorporated to effectively inject global scene motion, paving the way for comprehensive character art animation. Through extensive experiments, we demonstrate the effectiveness and generalizability of MikuDance across various character art and motion guidance, consistently producing high-quality animations with remarkable motion dynamics.

Related papers

• Animate-X: Universal Character Image Animation with Enhanced Motion Representation [42.73097432203482]
  Animate-X is a universal animation framework based on LDM for various character types, including anthropomorphic characters. We introduce the Pose Indicator, which captures comprehensive motion pattern from the driving video through both implicit and explicit manner. We also introduce a new Animated Anthropomorphic Benchmark to evaluate the performance of Animate-X on universal and widely applicable animation images.
  arXiv  Detail & Related papers  (2024-10-14T09:06:55Z)
• Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics [67.97235923372035]
  We present Puppet-Master, an interactive video generative model that can serve as a motion prior for part-level dynamics. At test time, given a single image and a sparse set of motion trajectories, Puppet-Master can synthesize a video depicting realistic part-level motion faithful to the given drag interactions.
  arXiv  Detail & Related papers  (2024-08-08T17:59:38Z)
• Towards High-Quality 3D Motion Transfer with Realistic Apparel Animation [69.36162784152584]
  We present a novel method aiming for high-quality motion transfer with realistic apparel animation. We propose a data-driven pipeline that learns to disentangle body and apparel deformations via two neural deformation modules. Our method produces results with superior quality for various types of apparel.
  arXiv  Detail & Related papers  (2024-07-15T22:17:35Z)
• Animate Your Motion: Turning Still Images into Dynamic Videos [58.63109848837741]
  We introduce Scene and Motion Conditional Diffusion (SMCD), a novel methodology for managing multimodal inputs. SMCD incorporates a recognized motion conditioning module and investigates various approaches to integrate scene conditions. Our design significantly enhances video quality, motion precision, and semantic coherence.
  arXiv  Detail & Related papers  (2024-03-15T10:36:24Z)
• MotionCrafter: One-Shot Motion Customization of Diffusion Models [66.44642854791807]
  We introduce MotionCrafter, a one-shot instance-guided motion customization method. MotionCrafter employs a parallel spatial-temporal architecture that injects the reference motion into the temporal component of the base model. During training, a frozen base model provides appearance normalization, effectively separating appearance from motion.
  arXiv  Detail & Related papers  (2023-12-08T16:31:04Z)
• Learning Motion Refinement for Unsupervised Face Animation [45.807582064277305]
  Unsupervised face animation aims to generate a human face video based on the appearance of a source image, mimicking the motion from a driving video. Existing methods typically adopted a prior-based motion model (e.g., the local affine motion model or the local thin-plate-spline motion model) In this work, we design a new unsupervised face animation approach to learn simultaneously the coarse and finer motions.
  arXiv  Detail & Related papers  (2023-10-21T05:52:25Z)
• TapMo: Shape-aware Motion Generation of Skeleton-free Characters [64.83230289993145]
  We present TapMo, a Text-driven Animation Pipeline for Motion in a broad spectrum of skeleton-free 3D characters. TapMo comprises two main components - Mesh Handle Predictor and Shape-aware Diffusion Module.
  arXiv  Detail & Related papers  (2023-10-19T12:14:32Z)
• Motion Puzzle: Arbitrary Motion Style Transfer by Body Part [6.206196935093063]
  Motion Puzzle is a novel motion style transfer network that advances the state-of-the-art in several important respects. Our framework extracts style features from multiple style motions for different body parts and transfers them locally to the target body parts. It can capture styles exhibited by dynamic movements, such as flapping and staggering, significantly better than previous work.
  arXiv  Detail & Related papers  (2022-02-10T19:56:46Z)
• Real-time Deep Dynamic Characters [95.5592405831368]
  We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing. We show that our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches.
  arXiv  Detail & Related papers  (2021-05-04T23:28:55Z)
• Self-supervised Motion Learning from Static Images [36.85209332144106]
  Motion from Static Images (MoSI) learns to encode motion information. MoSI can discover regions with large motion even without fine-tuning on the downstream datasets. We demonstrate that MoSI can discover regions with large motion even without fine-tuning on the downstream datasets.
  arXiv  Detail & Related papers  (2021-04-01T03:55:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.