Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraints

• URL: http://arxiv.org/abs/2411.19381v1
• Date: Thu, 28 Nov 2024 21:15:38 GMT
• Title: Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraints
• Authors: Gaurav Rai, Ojaswa Sharma,
• Abstract summary: We propose an approach for animating a given input sketch based on a descriptive text prompt. We leverage a pre-trained text-to-video diffusion model with SDS loss to guide the motion of the sketch's strokes. Our method surpasses state-of-the-art performance in both quantitative and qualitative evaluations.
• Score: 1.1510009152620668
• License:
• Abstract: Animating hand-drawn sketches using traditional tools is challenging and complex. Sketches provide a visual basis for explanations, and animating these sketches offers an experience of real-time scenarios. We propose an approach for animating a given input sketch based on a descriptive text prompt. Our method utilizes a parametric representation of the sketch's strokes. Unlike previous methods, which struggle to estimate smooth and accurate motion and often fail to preserve the sketch's topology, we leverage a pre-trained text-to-video diffusion model with SDS loss to guide the motion of the sketch's strokes. We introduce length-area (LA) regularization to ensure temporal consistency by accurately estimating the smooth displacement of control points across the frame sequence. Additionally, to preserve shape and avoid topology changes, we apply a shape-preserving As-Rigid-As-Possible (ARAP) loss to maintain sketch rigidity. Our method surpasses state-of-the-art performance in both quantitative and qualitative evaluations.

Related papers

• SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation [57.47730473674261]
  We introduce SwiftSketch, a model for image-conditioned vector sketch generation that can produce high-quality sketches in less than a second. SwiftSketch operates by progressively denoising stroke control points sampled from a Gaussian distribution. ControlSketch is a method that enhances SDS-based techniques by incorporating precise spatial control through a depth-aware ControlNet.
  arXiv  Detail & Related papers  (2025-02-12T18:57:12Z)
• VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Control [13.320911720001277]
  VidSketch is a method capable of generating high-quality video animations directly from any number of hand-drawn sketches and simple text prompts. Specifically, our method introduces a Level-Based Sketch Control Strategy to automatically the guidance strength of sketches adjust the generation process. A TempSpatial Attention mechanism is designed to enhance more consistency of generated video animations.
  arXiv  Detail & Related papers  (2025-02-03T06:45:00Z)
• Sketch-Guided Motion Diffusion for Stylized Cinemagraph Synthesis [15.988686454889823]
  Sketch2Cinemagraph is a sketch-guided framework that enables the conditional generation of stylized cinemagraphs from freehand sketches. We propose a novel latent motion diffusion model to estimate the motion field in the fluid regions of the generated landscape images.
  arXiv  Detail & Related papers  (2024-12-01T01:32:59Z)
• VIRES: Video Instance Repainting with Sketch and Text Guidance [46.24384664227624]
  We introduce VIRES, a video instance repainting method with sketch and text guidance. Existing approaches struggle with temporal consistency and accurate alignment with the provided sketch sequence. We propose the Sequential ControlNet with the standardized self-scaling. A sketch-aware encoder ensures that repainted results are aligned with the provided sketch sequence.
  arXiv  Detail & Related papers  (2024-11-25T08:55:41Z)
• FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations [65.64014682930164]
  Sketch animations offer a powerful medium for visual storytelling, from simple flip-book doodles to professional studio productions. We present FlipSketch, a system that brings back the magic of flip-book animation -- just draw your idea and describe how you want it to move!
  arXiv  Detail & Related papers  (2024-11-16T14:53:03Z)
• Sketch Video Synthesis [52.134906766625164]
  We propose a novel framework for sketching videos represented by the frame-wise B'ezier curve. Our method unlocks applications in sketch-based video editing and video doodling, enabled through video composition.
  arXiv  Detail & Related papers  (2023-11-26T14:14:04Z)
• Breathing Life Into Sketches Using Text-to-Video Priors [101.8236605955899]
  A sketch is one of the most intuitive and versatile tools humans use to convey their ideas visually. In this work, we present a method that automatically adds motion to a single-subject sketch. The output is a short animation provided in vector representation, which can be easily edited.
  arXiv  Detail & Related papers  (2023-11-21T18:09:30Z)
• Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening [58.09847349781176]
  We propose a novel deep learning method - Sketch-Aware Interpolation Network (SAIN) This approach incorporates multi-level guidance that formulates region-level correspondence, stroke-level correspondence and pixel-level dynamics. A multi-stream U-Transformer is then devised to characterize sketch inbetweening patterns using these multi-level guides through the integration of self / cross-attention mechanisms.
  arXiv  Detail & Related papers  (2023-08-25T09:51:03Z)
• SketchFFusion: Sketch-guided image editing with diffusion model [25.63913085329606]
  Sketch-guided image editing aims to achieve local fine-tuning of the image based on the sketch information provided by the user. We propose a sketch generation scheme that can preserve the main contours of an image and closely adhere to the actual sketch style drawn by the user.
  arXiv  Detail & Related papers  (2023-04-06T15:54:18Z)
• Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches [133.01690754567252]
  Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Deep Plastic Surgery is a novel, robust and controllable image editing framework that allows users to interactively edit images using hand-drawn sketch inputs.
  arXiv  Detail & Related papers  (2020-01-09T08:57:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.