VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Control

• URL: http://arxiv.org/abs/2502.01101v2
• Date: Mon, 17 Feb 2025 05:49:03 GMT
• Title: VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Control
• Authors: Lifan Jiang, Shuang Chen, Boxi Wu, Xiaotong Guan, Jiahui Zhang,
• Abstract summary: VidSketch is a method capable of generating high-quality video animations directly from any number of hand-drawn sketches and simple text prompts. Specifically, our method introduces a Level-Based Sketch Control Strategy to automatically the guidance strength of sketches adjust the generation process. A TempSpatial Attention mechanism is designed to enhance more consistency of generated video animations.
• Score: 13.320911720001277
• License:
• Abstract: With the advancement of generative artificial intelligence, previous studies have achieved the task of generating aesthetic images from hand-drawn sketches, fulfilling the public's needs for drawing. However, these methods are limited to static images and lack the ability to control video animation generation using hand-drawn sketches. To address this gap, we propose VidSketch, the first method capable of generating high-quality video animations directly from any number of hand-drawn sketches and simple text prompts, bridging the divide between ordinary users and professional artists. Specifically, our method introduces a Level-Based Sketch Control Strategy to automatically adjust the guidance strength of sketches during the generation process, accommodating users with varying drawing skills. Furthermore, a TempSpatial Attention mechanism is designed to enhance the spatiotemporal consistency of generated video animations, significantly improving the coherence across frames. You can find more detailed cases on our official website.

Related papers

• Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraints [1.1510009152620668]
  We propose an approach for animating a given input sketch based on a descriptive text prompt. We leverage a pre-trained text-to-video diffusion model with SDS loss to guide the motion of the sketch's strokes. Our method surpasses state-of-the-art performance in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2024-11-28T21:15:38Z)
• FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations [65.64014682930164]
  Sketch animations offer a powerful medium for visual storytelling, from simple flip-book doodles to professional studio productions. We present FlipSketch, a system that brings back the magic of flip-book animation -- just draw your idea and describe how you want it to move!
  arXiv  Detail & Related papers  (2024-11-16T14:53:03Z)
• Sketch Video Synthesis [52.134906766625164]
  We propose a novel framework for sketching videos represented by the frame-wise B'ezier curve. Our method unlocks applications in sketch-based video editing and video doodling, enabled through video composition.
  arXiv  Detail & Related papers  (2023-11-26T14:14:04Z)
• Breathing Life Into Sketches Using Text-to-Video Priors [101.8236605955899]
  A sketch is one of the most intuitive and versatile tools humans use to convey their ideas visually. In this work, we present a method that automatically adds motion to a single-subject sketch. The output is a short animation provided in vector representation, which can be easily edited.
  arXiv  Detail & Related papers  (2023-11-21T18:09:30Z)
• Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening [58.09847349781176]
  We propose a novel deep learning method - Sketch-Aware Interpolation Network (SAIN) This approach incorporates multi-level guidance that formulates region-level correspondence, stroke-level correspondence and pixel-level dynamics. A multi-stream U-Transformer is then devised to characterize sketch inbetweening patterns using these multi-level guides through the integration of self / cross-attention mechanisms.
  arXiv  Detail & Related papers  (2023-08-25T09:51:03Z)
• DeepFacePencil: Creating Face Images from Freehand Sketches [77.00929179469559]
  Existing image-to-image translation methods require a large-scale dataset of paired sketches and images for supervision. We propose DeepFacePencil, an effective tool that is able to generate photo-realistic face images from hand-drawn sketches.
  arXiv  Detail & Related papers  (2020-08-31T03:35:21Z)
• Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches [133.01690754567252]
  Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Deep Plastic Surgery is a novel, robust and controllable image editing framework that allows users to interactively edit images using hand-drawn sketch inputs.
  arXiv  Detail & Related papers  (2020-01-09T08:57:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.