SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models

• URL: http://arxiv.org/abs/2508.07149v1
• Date: Sun, 10 Aug 2025 02:45:59 GMT
• Title: SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models
• Authors: Ruolin Yang, Da Li, Honggang Zhang, Yi-Zhe Song,
• Abstract summary: We propose a novel sketch animation model SketchAnimator, which enables adding creative motion to a given sketch, like "a jumping car"<n>In stages 1 and 2, we utilize LoRA to integrate sketch appearance information and motion dynamics from the reference video into the pre-trained T2V model.<n>In the third stage, we utilize Score Distillation Sampling (SDS) to update the parameters of the Bezier curves in each sketch frame according to the acquired motion information.
• Score: 52.15095222622447
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sketching is a uniquely human tool for expressing ideas and creativity. The animation of sketches infuses life into these static drawings, opening a new dimension for designers. Animating sketches is a time-consuming process that demands professional skills and extensive experience, often proving daunting for amateurs. In this paper, we propose a novel sketch animation model SketchAnimator, which enables adding creative motion to a given sketch, like "a jumping car''. Namely, given an input sketch and a reference video, we divide the sketch animation into three stages: Appearance Learning, Motion Learning and Video Prior Distillation. In stages 1 and 2, we utilize LoRA to integrate sketch appearance information and motion dynamics from the reference video into the pre-trained T2V model. In the third stage, we utilize Score Distillation Sampling (SDS) to update the parameters of the Bezier curves in each sketch frame according to the acquired motion information. Consequently, our model produces a sketch video that not only retains the original appearance of the sketch but also mirrors the dynamic movements of the reference video. We compare our method with alternative approaches and demonstrate that it generates the desired sketch video under the challenge of one-shot motion customization.

Related papers

• 4-Doodle: Text to 3D Sketches that Move! [60.89021458068987]
  4-Doodle is the first training-free framework for generating dynamic 3D sketches from text.<n>Our method produces temporally realistic and structurally stable 3D sketch animations, outperforming existing baselines in both fidelity and controllability.
  arXiv  Detail & Related papers  (2025-10-29T09:33:29Z)
• DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds [64.53681498600065]
  DreamDance is an animation framework capable of producing stable, consistent character and scene motion conditioned on precise camera trajectories.<n>We train a pose-aware video inpainting model that injects the dynamic character into the scene video while enhancing background quality.
  arXiv  Detail & Related papers  (2025-05-30T15:54:34Z)
• Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation [22.325990468075368]
  Animators use the 2D sketches in storyboards as references to craft the desired 3D animations through a trial-and-error process.<n>There is a high demand for automated methods that can directly translate 2D storyboard sketches into 3D animations.<n>We present Sketch2Anim, composed of two key modules for sketch constraint understanding and motion generation.
  arXiv  Detail & Related papers  (2025-04-27T10:38:17Z)
• Multi-Object Sketch Animation by Scene Decomposition and Motion Planning [9.124628743276691]
  MoSketch is a training-data-free method for animating multi-object sketches.<n>MoSketch has four novel modules, i.e., LLM-based scene decomposition, LLM-based motion planning, multi-grained motion refinement, and compositional SDS.<n>MoSketch takes a pioneering step towards multi-object sketch animation, opening new avenues for future research and applications.
  arXiv  Detail & Related papers  (2025-03-25T05:00:11Z)
• FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations [65.64014682930164]
  Sketch animations offer a powerful medium for visual storytelling, from simple flip-book doodles to professional studio productions. We present FlipSketch, a system that brings back the magic of flip-book animation -- just draw your idea and describe how you want it to move!
  arXiv  Detail & Related papers  (2024-11-16T14:53:03Z)
• Breathing Life Into Sketches Using Text-to-Video Priors [101.8236605955899]
  A sketch is one of the most intuitive and versatile tools humans use to convey their ideas visually. In this work, we present a method that automatically adds motion to a single-subject sketch. The output is a short animation provided in vector representation, which can be easily edited.
  arXiv  Detail & Related papers  (2023-11-21T18:09:30Z)
• Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening [58.09847349781176]
  We propose a novel deep learning method - Sketch-Aware Interpolation Network (SAIN) This approach incorporates multi-level guidance that formulates region-level correspondence, stroke-level correspondence and pixel-level dynamics. A multi-stream U-Transformer is then devised to characterize sketch inbetweening patterns using these multi-level guides through the integration of self / cross-attention mechanisms.
  arXiv  Detail & Related papers  (2023-08-25T09:51:03Z)
• SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches [0.9645196221785693]
  2D animation is a common factor in game development, used for characters, effects and background art. Automated animation approaches exist, but are designed without animators in mind. We propose a problem formulation that adheres more closely to the standard workflow of animation.
  arXiv  Detail & Related papers  (2022-09-01T02:43:19Z)
• Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt [125.17887147597567]
  We present a model of learning Sketch BiBERT Representation from Transformer (Sketch-BERT) We generalize BERT to sketch domain, with the novel proposed components and pre-training algorithms. We show that the learned representation of Sketch-BERT can help and improve the performance of the downstream tasks of sketch recognition, sketch retrieval, and sketch gestalt.
  arXiv  Detail & Related papers  (2020-05-19T01:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.