Breathing Life Into Sketches Using Text-to-Video Priors
- URL: http://arxiv.org/abs/2311.13608v1
- Date: Tue, 21 Nov 2023 18:09:30 GMT
- Title: Breathing Life Into Sketches Using Text-to-Video Priors
- Authors: Rinon Gal, Yael Vinker, Yuval Alaluf, Amit H. Bermano, Daniel
Cohen-Or, Ariel Shamir, Gal Chechik
- Abstract summary: A sketch is one of the most intuitive and versatile tools humans use to convey their ideas visually.
In this work, we present a method that automatically adds motion to a single-subject sketch.
The output is a short animation provided in vector representation, which can be easily edited.
- Score: 101.8236605955899
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: A sketch is one of the most intuitive and versatile tools humans use to
convey their ideas visually. An animated sketch opens another dimension to the
expression of ideas and is widely used by designers for a variety of purposes.
Animating sketches is a laborious process, requiring extensive experience and
professional design skills. In this work, we present a method that
automatically adds motion to a single-subject sketch (hence, "breathing life
into it"), merely by providing a text prompt indicating the desired motion. The
output is a short animation provided in vector representation, which can be
easily edited. Our method does not require extensive training, but instead
leverages the motion prior of a large pretrained text-to-video diffusion model
using a score-distillation loss to guide the placement of strokes. To promote
natural and smooth motion and to better preserve the sketch's appearance, we
model the learned motion through two components. The first governs small local
deformations and the second controls global affine transformations.
Surprisingly, we find that even models that struggle to generate sketch videos
on their own can still serve as a useful backbone for animating abstract
representations.
Related papers
- VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Control [13.320911720001277]
VidSketch is a method capable of generating high-quality video animations directly from any number of hand-drawn sketches and simple text prompts.
Specifically, our method introduces a Level-Based Sketch Control Strategy to automatically the guidance strength of sketches adjust the generation process.
A TempSpatial Attention mechanism is designed to enhance more consistency of generated video animations.
arXiv Detail & Related papers (2025-02-03T06:45:00Z) - AniDoc: Animation Creation Made Easier [54.97341104616779]
Our research focuses on reducing the labor costs in the production of 2D animation by harnessing the potential of increasingly powerful AI.
AniDoc emerges as a video line art colorization tool, which automatically converts sketch sequences into colored animations.
Our model exploits correspondence matching as an explicit guidance, yielding strong robustness to the variations between the reference character and each line art frame.
arXiv Detail & Related papers (2024-12-18T18:59:59Z) - Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraints [1.1510009152620668]
We propose an approach for animating a given input sketch based on a descriptive text prompt.
We leverage a pre-trained text-to-video diffusion model with SDS loss to guide the motion of the sketch's strokes.
Our method surpasses state-of-the-art performance in both quantitative and qualitative evaluations.
arXiv Detail & Related papers (2024-11-28T21:15:38Z) - FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations [65.64014682930164]
Sketch animations offer a powerful medium for visual storytelling, from simple flip-book doodles to professional studio productions.
We present FlipSketch, a system that brings back the magic of flip-book animation -- just draw your idea and describe how you want it to move!
arXiv Detail & Related papers (2024-11-16T14:53:03Z) - AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment [64.02822911038848]
We present AnimateZoo, a zero-shot diffusion-based video generator to produce animal animations.
Key technique used in our AnimateZoo is subject alignment, which includes two steps.
Our model is capable of generating videos characterized by accurate movements, consistent appearance, and high-fidelity frames.
arXiv Detail & Related papers (2024-04-07T12:57:41Z) - AnimateZero: Video Diffusion Models are Zero-Shot Image Animators [63.938509879469024]
We propose AnimateZero to unveil the pre-trained text-to-video diffusion model, i.e., AnimateDiff.
For appearance control, we borrow intermediate latents and their features from the text-to-image (T2I) generation.
For temporal control, we replace the global temporal attention of the original T2V model with our proposed positional-corrected window attention.
arXiv Detail & Related papers (2023-12-06T13:39:35Z) - SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation [111.2195741547517]
We present a method to generate controlled sketches using a text-conditioned diffusion model trained on pixel representations of images.
Our objective is to empower non-professional users to create sketches and, through a series of optimisation processes, transform a narrative into a storyboard.
arXiv Detail & Related papers (2023-08-27T19:44:44Z) - SketchBetween: Video-to-Video Synthesis for Sprite Animation via
Sketches [0.9645196221785693]
2D animation is a common factor in game development, used for characters, effects and background art.
Automated animation approaches exist, but are designed without animators in mind.
We propose a problem formulation that adheres more closely to the standard workflow of animation.
arXiv Detail & Related papers (2022-09-01T02:43:19Z) - Sketch Me A Video [32.38205496481408]
We introduce a new video synthesis task by employing two rough bad-drwan sketches only as input to create a realistic portrait video.
A two-stage Sketch-to-Video model is proposed, which consists of two key novelties.
arXiv Detail & Related papers (2021-10-10T05:40:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.