Text-Guided Synthesis of Eulerian Cinemagraphs
- URL: http://arxiv.org/abs/2307.03190v3
- Date: Tue, 26 Sep 2023 02:46:02 GMT
- Title: Text-Guided Synthesis of Eulerian Cinemagraphs
- Authors: Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Lee, Sergey
Tulyakov, Jun-Yan Zhu
- Abstract summary: We introduce Text2Cinemagraph, a fully automated method for creating cinemagraphs from text descriptions.
We focus on cinemagraphs of fluid elements, such as flowing rivers, and drifting clouds, which exhibit continuous motion and repetitive textures.
- Score: 81.20353774053768
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We introduce Text2Cinemagraph, a fully automated method for creating
cinemagraphs from text descriptions - an especially challenging task when
prompts feature imaginary elements and artistic styles, given the complexity of
interpreting the semantics and motions of these images. We focus on
cinemagraphs of fluid elements, such as flowing rivers, and drifting clouds,
which exhibit continuous motion and repetitive textures. Existing single-image
animation methods fall short on artistic inputs, and recent text-based video
methods frequently introduce temporal inconsistencies, struggling to keep
certain regions static. To address these challenges, we propose an idea of
synthesizing image twins from a single text prompt - a pair of an artistic
image and its pixel-aligned corresponding natural-looking twin. While the
artistic image depicts the style and appearance detailed in our text prompt,
the realistic counterpart greatly simplifies layout and motion analysis.
Leveraging existing natural image and video datasets, we can accurately segment
the realistic image and predict plausible motion given the semantic
information. The predicted motion can then be transferred to the artistic image
to create the final cinemagraph. Our method outperforms existing approaches in
creating cinemagraphs for natural landscapes as well as artistic and
other-worldly scenes, as validated by automated metrics and user studies.
Finally, we demonstrate two extensions: animating existing paintings and
controlling motion directions using text.
Related papers
- CLII: Visual-Text Inpainting via Cross-Modal Predictive Interaction [23.683636588751753]
State-of-the-art inpainting methods are mainly designed for nature images and cannot correctly recover text within scene text images.
We identify the visual-text inpainting task to achieve high-quality scene text image restoration and text completion.
arXiv Detail & Related papers (2024-07-23T06:12:19Z) - Intelligent Artistic Typography: A Comprehensive Review of Artistic Text Design and Generation [15.367944842667146]
Artistic text generation aims to amplify the aesthetic qualities of text while maintaining readability.
Artistic text stylization concentrates on the text effect overlaid upon the text, such as shadows, outlines, colors, glows, and textures.
Stylistization focuses on the deformation of the characters to strengthen their visual representation by mimicking the semantic understanding within the text.
arXiv Detail & Related papers (2024-07-20T06:45:09Z) - Dynamic Typography: Bringing Text to Life via Video Diffusion Prior [73.72522617586593]
We present an automated text animation scheme, termed "Dynamic Typography"
It deforms letters to convey semantic meaning and infuses them with vibrant movements based on user prompts.
Our technique harnesses vector graphics representations and an end-to-end optimization-based framework.
arXiv Detail & Related papers (2024-04-17T17:59:55Z) - LivePhoto: Real Image Animation with Text-guided Motion Control [51.31418077586208]
This work presents a practical system, named LivePhoto, which allows users to animate an image of their interest with text descriptions.
We first establish a strong baseline that helps a well-learned text-to-image generator (i.e., Stable Diffusion) take an image as a further input.
We then equip the improved generator with a motion module for temporal modeling and propose a carefully designed training pipeline to better link texts and motions.
arXiv Detail & Related papers (2023-12-05T17:59:52Z) - DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors [63.43133768897087]
We propose a method to convert open-domain images into animated videos.
The key idea is to utilize the motion prior to text-to-video diffusion models by incorporating the image into the generative process as guidance.
Our proposed method can produce visually convincing and more logical & natural motions, as well as higher conformity to the input image.
arXiv Detail & Related papers (2023-10-18T14:42:16Z) - Text2Performer: Text-Driven Human Video Generation [97.3849869893433]
Text-driven content creation has evolved to be a transformative technique that revolutionizes creativity.
Here we study the task of text-driven human video generation, where a video sequence is synthesized from texts describing the appearance and motions of a target performer.
In this work, we present Text2Performer to generate vivid human videos with articulated motions from texts.
arXiv Detail & Related papers (2023-04-17T17:59:02Z) - Text to artistic image generation [0.0]
We create an end-to-end solution that can generate artistic images from text descriptions.
Due to the lack of datasets with paired text description and artistic images, it is hard to directly train an algorithm which can create art based on text input.
arXiv Detail & Related papers (2022-05-05T04:44:56Z) - Animating Pictures with Eulerian Motion Fields [90.30598913855216]
We show a fully automatic method for converting a still image into a realistic animated looping video.
We target scenes with continuous fluid motion, such as flowing water and billowing smoke.
We propose a novel video looping technique that flows features both forward and backward in time and then blends the results.
arXiv Detail & Related papers (2020-11-30T18:59:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.