Breathing Life Into Sketches Using Text-to-Video Priors
- URL: http://arxiv.org/abs/2311.13608v1
- Date: Tue, 21 Nov 2023 18:09:30 GMT
- Title: Breathing Life Into Sketches Using Text-to-Video Priors
- Authors: Rinon Gal, Yael Vinker, Yuval Alaluf, Amit H. Bermano, Daniel
Cohen-Or, Ariel Shamir, Gal Chechik
- Abstract summary: A sketch is one of the most intuitive and versatile tools humans use to convey their ideas visually.
In this work, we present a method that automatically adds motion to a single-subject sketch.
The output is a short animation provided in vector representation, which can be easily edited.
- Score: 101.8236605955899
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: A sketch is one of the most intuitive and versatile tools humans use to
convey their ideas visually. An animated sketch opens another dimension to the
expression of ideas and is widely used by designers for a variety of purposes.
Animating sketches is a laborious process, requiring extensive experience and
professional design skills. In this work, we present a method that
automatically adds motion to a single-subject sketch (hence, "breathing life
into it"), merely by providing a text prompt indicating the desired motion. The
output is a short animation provided in vector representation, which can be
easily edited. Our method does not require extensive training, but instead
leverages the motion prior of a large pretrained text-to-video diffusion model
using a score-distillation loss to guide the placement of strokes. To promote
natural and smooth motion and to better preserve the sketch's appearance, we
model the learned motion through two components. The first governs small local
deformations and the second controls global affine transformations.
Surprisingly, we find that even models that struggle to generate sketch videos
on their own can still serve as a useful backbone for animating abstract
representations.
Related papers
- AniClipart: Clipart Animation with Text-to-Video Priors [28.76809141136148]
We introduce AniClipart, a system that transforms static images into high-quality motion sequences guided by text-to-video priors.
Experimental results show that the proposed AniClipart consistently outperforms existing image-to-video generation models.
arXiv Detail & Related papers (2024-04-18T17:24:28Z) - AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment [64.02822911038848]
We present AnimateZoo, a zero-shot diffusion-based video generator to produce animal animations.
Key technique used in our AnimateZoo is subject alignment, which includes two steps.
Our model is capable of generating videos characterized by accurate movements, consistent appearance, and high-fidelity frames.
arXiv Detail & Related papers (2024-04-07T12:57:41Z) - Iterative Motion Editing with Natural Language [23.836693717924845]
Text-to-motion diffusion models can generate realistic animations from text prompts, but do not support fine-grained motion editing controls.
We present a method for using natural language to iteratively specify local edits to existing character animations.
Our system can perform motion edits that respect the animator's editing intent, remain faithful to the original animation, and yield realistic character animation results.
arXiv Detail & Related papers (2023-12-15T22:38:24Z) - AnimateZero: Video Diffusion Models are Zero-Shot Image Animators [63.938509879469024]
We propose AnimateZero to unveil the pre-trained text-to-video diffusion model, i.e., AnimateDiff.
For appearance control, we borrow intermediate latents and their features from the text-to-image (T2I) generation.
For temporal control, we replace the global temporal attention of the original T2V model with our proposed positional-corrected window attention.
arXiv Detail & Related papers (2023-12-06T13:39:35Z) - SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation [111.2195741547517]
We present a method to generate controlled sketches using a text-conditioned diffusion model trained on pixel representations of images.
Our objective is to empower non-professional users to create sketches and, through a series of optimisation processes, transform a narrative into a storyboard.
arXiv Detail & Related papers (2023-08-27T19:44:44Z) - CLIPascene: Scene Sketching with Different Types and Levels of
Abstraction [48.30702300230904]
We present a method for converting a given scene image into a sketch using different types and multiple levels of abstraction.
The first considers the fidelity of the sketch, varying its representation from a more precise portrayal of the input to a looser depiction.
The second is defined by the visual simplicity of the sketch, moving from a detailed depiction to a sparse sketch.
arXiv Detail & Related papers (2022-11-30T18:54:32Z) - SketchBetween: Video-to-Video Synthesis for Sprite Animation via
Sketches [0.9645196221785693]
2D animation is a common factor in game development, used for characters, effects and background art.
Automated animation approaches exist, but are designed without animators in mind.
We propose a problem formulation that adheres more closely to the standard workflow of animation.
arXiv Detail & Related papers (2022-09-01T02:43:19Z) - I Know What You Draw: Learning Grasp Detection Conditioned on a Few
Freehand Sketches [74.63313641583602]
We propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects.
Our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications.
arXiv Detail & Related papers (2022-05-09T04:23:36Z) - Sketch Me A Video [32.38205496481408]
We introduce a new video synthesis task by employing two rough bad-drwan sketches only as input to create a realistic portrait video.
A two-stage Sketch-to-Video model is proposed, which consists of two key novelties.
arXiv Detail & Related papers (2021-10-10T05:40:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.