Sketch2Motion: Text-driven 2D Sketch to 3D Animation via Diffusion-guided Skeleton Optimization

• URL: http://arxiv.org/abs/2605.28394v1
• Date: Wed, 27 May 2026 12:32:07 GMT
• Title: Sketch2Motion: Text-driven 2D Sketch to 3D Animation via Diffusion-guided Skeleton Optimization
• Abstract summary: Animation of 2D hand-drawn sketches provides an effective medium for visual communication.<n>Recent approaches to converting 2D sketches to 3D animations have mainly focused on specific types of motion.<n>We propose Sketch2Motion, a diffusion-guided framework for skeleton-based motion synthesis.
• Score: 0.6445605125467574
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Animation of 2D hand-drawn sketches provides an effective medium for visual communication. However, these sketches pose challenges, particularly in handling occlusions and accurately mapping motion. While 3D animation naturally addresses these challenges, estimating 3D motion remains a very complex task. Recent approaches to converting 2D sketches to 3D animations have mainly focused on specific types of motion, such as bipedal movements and facial expressions. We propose Sketch2Motion, a diffusion-guided framework for skeleton-based motion synthesis that combines classical character animation pipelines with deep generative priors. Our method represents motion using skeletal transformations, which are propagated to mesh deformations via linear blend skinning. To guide the resulting animation toward realistic and semantically meaningful motion, we integrate a text-to-video diffusion model via motion-aware score-distillation sampling (MoSDS), enabling optimization without paired motion data. Additionally, we apply physics-inspired smoothness, topological, and contact constraints to stabilize optimization and preserve motion plausibility. Further, we integrate a spring-mass simulator to introduce secondary motion effects. The proposed framework is generalized, fully differentiable, modular, and compatible with biped, quadruped, and non-living articulated characters. Experiments demonstrate that our approach produces temporally coherent, text-aligned animations that outperform baseline motion transfer methods that lack generative priors or explicit physical constraints. We will make our code and dataset publicly available.

Related papers

• IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation [58.297199313494]
  Implicit methods capture motion semantics directly from driving video, but suffer from identity leakage and entanglement between motion and appearance.<n>We propose a novel implicit motion representation that compresses per-frame motion into compact 1D motion tokens.<n>Our methodology employs a three-stage training strategy to enhance the training efficiency and ensure high fidelity.
  arXiv  Detail & Related papers  (2026-02-07T11:17:20Z)
• 4-Doodle: Text to 3D Sketches that Move! [60.89021458068987]
  4-Doodle is the first training-free framework for generating dynamic 3D sketches from text.<n>Our method produces temporally realistic and structurally stable 3D sketch animations, outperforming existing baselines in both fidelity and controllability.
  arXiv  Detail & Related papers  (2025-10-29T09:33:29Z)
• AnimaX: Animating the Inanimate in 3D with Joint Video-Pose Diffusion Models [24.410731608387238]
  AnimaX is a feed-forward 3D animation framework that bridges the motion priors of video diffusion models with the controllable structure of skeleton-based animation.<n>Our method represents 3D motion as multi-view, multi-frame 2D pose maps, and enables joint video-pose diffusion conditioned on template renderings.
  arXiv  Detail & Related papers  (2025-06-24T17:59:58Z)
• Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation [22.325990468075368]
  Animators use the 2D sketches in storyboards as references to craft the desired 3D animations through a trial-and-error process.<n>There is a high demand for automated methods that can directly translate 2D storyboard sketches into 3D animations.<n>We present Sketch2Anim, composed of two key modules for sketch constraint understanding and motion generation.
  arXiv  Detail & Related papers  (2025-04-27T10:38:17Z)
• Animating the Uncaptured: Humanoid Mesh Animation with Video Diffusion Models [71.78723353724493]
  Animation of humanoid characters is essential in various graphics applications.<n>We propose an approach to synthesize 4D animated sequences of input static 3D humanoid meshes.
  arXiv  Detail & Related papers  (2025-03-20T10:00:22Z)
• FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations [65.64014682930164]
  Sketch animations offer a powerful medium for visual storytelling, from simple flip-book doodles to professional studio productions. We present FlipSketch, a system that brings back the magic of flip-book animation -- just draw your idea and describe how you want it to move!
  arXiv  Detail & Related papers  (2024-11-16T14:53:03Z)
• TapMo: Shape-aware Motion Generation of Skeleton-free Characters [64.83230289993145]
  We present TapMo, a Text-driven Animation Pipeline for Motion in a broad spectrum of skeleton-free 3D characters. TapMo comprises two main components - Mesh Handle Predictor and Shape-aware Diffusion Module.
  arXiv  Detail & Related papers  (2023-10-19T12:14:32Z)
• SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches [0.9645196221785693]
  2D animation is a common factor in game development, used for characters, effects and background art. Automated animation approaches exist, but are designed without animators in mind. We propose a problem formulation that adheres more closely to the standard workflow of animation.
  arXiv  Detail & Related papers  (2022-09-01T02:43:19Z)
• Deep Animation Video Interpolation in the Wild [115.24454577119432]
  In this work, we formally define and study the animation video code problem for the first time. We propose an effective framework, AnimeInterp, with two dedicated modules in a coarse-to-fine manner. Notably, AnimeInterp shows favorable perceptual quality and robustness for animation scenarios in the wild.
  arXiv  Detail & Related papers  (2021-04-06T13:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.